Caching data from web data sources, such as image servers, using persistent cache in a Manifold project can significantly improve the performance of web data sources, albeit at the cost of some storage space. In web sites where the same views are frequently visited, such as projects that have locations set which most visitors to the web site will use, the performance increase can be dramatic compared to having to wait for a site such as OSM to re-send data.
A related topic is how cache is used with Manifold running as a client when connecting to WMS data sources. See the discussion (and illustrations) in the Display Sharpness in WMS Clients section of the Server: Publish Web Maps topic.
Manifold projects opened by Server, when running either as a TCP server or as an HTTP server, are read-only. When serving data from external data sources via Server, either as a TCP server or an HTTP server, that limits the ability of the system to use persistent cache to cache data provided by external data source, since the read-only project cannot be modified to save data into cache within the project.
We can get around that limitation by using an external database such as PostgreSQL or SQLite to host the web data source and its cache, with Server automatically tunneling through the external database data sourcde to the web data source. As this topic illustrates, a simple file database like SQLite is easy to use in such applications. Using SQLite avoids the administrative effort of standing up a PostgreSQL database installation.
Let us first review how a new data source is created within the .map project. If we open a new Manifold project and create a data source, such as a web data source like an OpenStreet Maps base image server, that new data source will be created within the Manifold .map project.
An option in the New Data Source dialog is to check the Save cached data between sessions box.
Checking that box will create the data source and will also create cache tables within the .map project's System Data folder. The imageserver_cache_openstreetbase table saves tiles that have been viewed as we pan and zoom within maps that include the OpenStreet Maps Base Image layer. Because those tiles are saved in the .map project, they can be fetched much faster from Manifold native storage than re-fetching them over the web from OSM's server should we ever return to a view at the same zoom level that includes those tiles. See the discussion in the Cache section of the Web Servers topic.
All that is fine and very useful, but it cannot work if the the .map project file has been opened read-only, because the imageserver_cache_openstreetbase table cannot be updated with new files.
We can get around that limitation by first creating a data source that is a read/write database (such as PostgreSQL or SQLite), and then creating the image server data source within that read/write database. Surprisingly, Manifold can create web data sources such as image servers within other databases. If those other databases are read/write, then the cache table for the web data source can be created within the read/write database, and that will allow many users of a Manifold Server instance to share those cache tables, updating them as users browse or otherwise utilize the data source that was created within the database.
Let us now take a look at first linking in an extermal database as a data source, and then creating a web data source within that external data source.
In this example we will create an SQLite database and then we link that SQLite database into our project. Most people think of PostgreSQL or similar enterprice class databases when setting up a multiuser, read/write project involving Manifold, but SQLite is also read/write with multiple users and can be used when the exceptional speed of PostgreSQL or similar is not required. Using SQLite to host a cache can still be very much faster than repeatedly pulling the same data from a web data source.
The easiest way to create a blank SQLite database is to first create a new, blank table in Manifold. We then right-click on that table and choose Export, exporting the table into an SQLite file database. In this example we create a blank table and then we export it into an SQLite file called OSM_cache.sqlite.
In a new, blank .map project we choose File - Link and we link the OSM_cache.sqlite file into the project. That creates a new data source called OSM_cache which is read/write by default. The new data source contains a single table, the blank Table we created earlier.
We click the Table to make sure the focus in the Project is within the OSM_cache data source we created. We then choose File-Create - New Data Source to create an OSM image server:
We name the new data source OSM Base, and we make sure to check the Save cached data between sessions box.
Press Create Data Source.
The new image server data source is created within the OSM_cache data source, that is, it is created within the SQLite database. It is a fully read/write data source, and the cache files that have been created within it are fully read/write by multiple simultaneous users.
We can create a new, blank map and then drag the OpenStreet Maps Base Image layer into the map. We can then pan and zoom within the map to whatever view we want.
In the example above, we have panned and zoomed into Paris, for a closer look at the Louvre.
We click the Map component to move the focus into the local, .map project and then we use the Locations button in the main toolbar to save the current location. We rename that location to Louvre.
We pan and zoom to a few more locations of interest, and we save those locations with descriptive names. In the illustration above we have collapsed the OSM_cache data source and the OSM Base datasource within, to provide more room in the illustration of the Project pane to show the locations we have created.
Next, we serve the project to the web, serving the Map component using Manifold Server as an HTTP server. We can visit the web site thus created from a browser:
The web page opens by default zoomed to fit to the only layer in the Map, the OpenStreet Map Base layer. From the Locations button in the web site's toolbar, we choose Louvre.
The view instantly appears, since the tiles for the view are already in the cache for the OSM Base data source. They were saved there, within the read/write SQLite database file, when we had the project open in the desktop and we panned and zoomed to the location of the Louvre in Paris to create that saved location.
We can choose another location from the Locations button, such as Chartres.
Again, the view instantly appears since the tiles for that location are also in the cache. There is no need to fetch them from the OSM servers. Instead, they are served directly from within the SQLite database file that hosts the OSM Base data source and the cache for that data source.
If other users on other machines visit the web site we have created, whenever they pan or zoom the view the tiles fetched from OSM will be cached within the SQLite database file in which we have created the OSM Base data source. If we have many users who are actively viewing a large geographic area with many different views at many different zoom levels, the cache can become very large, increasing the size of the SQLite file greatly, so we should make sure to have plenty of free space on disk on the machine where Server is running.
We can see the SQLite file database is being used for storing cache by visiting the folder where the SQLite file is located and noting the increasing size of the file.
In the illustration above we have browed the web site, panning and zooming to different locations. We note the SQLite file size is 112,660 KB.
After an additional minute or two of panning and zooming to new locations, we see that the size of the SQLite file has increased to 158,324 KB. File size can increase quickly if the regions browsed are large and many users browse to new locations and new zooms. If our map shows a limited region, such as a single town or city, as visitors browse the site they will more and more frequently view the same places at the same zoom levels, thus reusing the same tiles. After a while the size of the file used to save the cache will reach a plateau as almost all of the tiles available from OSM for that region will be in cache.
SQLite is a popular solution for back-end databases for low to medium traffic websites (most websites). The sqlite.org page advises that "Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite. The 100K hits/day figure is a conservative estimate, not a hard upper bound."
If we have very popular websites, we probably would use a faster database than SQLite. For example, we could stand up a PostgreSQL instance and create the OSM Base database within the PostgreSQL database. That would work well when many thousands of people browse a web site we are serving with Server, in cases where such browsing ends up putting hundreds of gigabytes of OSM tiles into cache.
Even though it is slower than a really fast database like PostgreSQL, SQLite nonetheless can be much faster than waiting for a slower image server like OSM to serve tiles. SQLite can work well for web sites that serve relatively local data, for example, for a single town or county, where there is much repetitive viewing of locations. Most users in such sites will jump directly to a particular location, so many views will be repeated in terms of geographic location and zoom. That results in a relatively limited amount of image data being cached, so the SQLite database will remain relatively limited in size without growing to hundreds of gigabutes or more.
Credentials encryption - Credentials are stored in database and other data sources in plain text. Manifold expects to provide encryption options so data sources can be served by Server without exposing credentials.
TCP - Manifold clients connect to Server running as a TCP server through TCP protocol, a standard part of networking.
Exclusive use - A Manifold project being served by Server cannot be opened or used by a different Server session or by a regular Manifold desktop session. To update that project, the Server session using it must be shut down. Server cannot share a project that is in use in a regular Manifold desktop session or which is being served by a different Server session.
Exclusive binding - For security, Server binds to the network interface in exclusive mode.
Console title - When launched in a Command Prompt window, Server sets the console title to Server - addr:port - filename. This hides sensitive information like user logins and passwords and it also shows connection information even if that was not spelled out in the command line. The console window will show the server type (HTTP or TCP) in the window caption when Server is run in a console window.
Non-routable IP addresses - The examples in this documentation use non-routable IP addresses beginning with 192.... Such IP addresses may be visible on your local network, but they cannot be visited across Internet. When connecting to Server through the Internet, either for sharing data using the TCP server or for publishing maps using the HTTP server, the computer should have a static IP address that is routable (visible over Internet), like 20.81.111.85, and the Server instances that are launched should use that IP address.
File - Create - New Data Source