agnos.is

Published: 2024-09-23T22:16:34+02:00

In the latest versions of the OpenStreetMap tool for OpenWebUI (0.6.3+), I have added proper handling of real-time location. OpenWebUI has the ability to report the user's location via the browser geolocation API.

https://git.agnos.is/projectmoon/open-webui-filters

Previous versions of the OSM tool did not prompt the large language model (LLM) properly when this information was available, so results for questions like “Where is the nearest grocery store?” would either be completely nonsensical, or centered on some nearby major city that the LLM statistically associates with the GPS coordinates reported by the browser.

Proper Handling of Real-Time Location

In order to have the OSM tool be able to answer questions like “where is the nearest grocery store to me?“, it needs access to your realtime location. This can be accomplished with the following steps: – Enable user location in your user settings. – Create a model with a system prompt that references the variable {{USER_LOCATION}}. – OpenWebUI will automatically substitute the GPS coordinates reported by the browser into the model's system prompt on every message.

Without the {{USER_LOCATION}} variable in the system prompt, OpenWebUI will not be able to give your real-time location to the LLM, and the OpenStreetMap tool will not give accurate results.

How Does It Work?

This was quite simple to enable. OpenWebUI uses the docstrings of Python functions to assemble a “tools schema” for the language model to use. OSM tool prior to 0.6.3 generated a schema that told the LLM to search for things near an “address or place.” Because of this, models would dutifully give a place or address, and NOT GPS coordinates.

This resulted in interesting behavior. The model would pick a city that it statistically associates with the user's GPS coordinates, and feed that in as a place to search nearby.

A simple tweak fixes this. The schema now tells the model to search for a “place, address, or GPS coordinates.” Models will now give the OSM tool GPS coordinates when available, and thus the tool can now answer questions like “Where is the nearest grocery store?”

License: CC-BY-SA-4.0.

Written by: @[email protected]