Custom Servers

When choosing a Type of Webserver: imageserver, we can choose the Custom subtype to utilize a very wide range of web servers that provide image tiles using connection strings that contain some more-or-less typically encountered, but far from standard, control sequences.    Control sequences are tokens within curly { } brackets that are substituted during the action of the server with values that control the result.    To learn how a specific server uses control sequences, as well as learning other required information such as the tile size used and the zoom range supported, we must consult whatever documentation the server operator has provided.  

 

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose Web Server: imageserver and Custom in the secondary box below.

Source

A URL connection string that contains control sequences used by the server.  

 Browse button

Click to help populate the Source box.  Clicking the [...] Browse button will launch the Web Login dialog, to allow use of a login and password plus use of a proxy server if desired.   The Web Login dialog is also handy for providing a Test button that can be used to test the connection.

Open as read-only

Open the data source read-only.  Usually has no effect with web servers since they are read-only in any event.

Cache data

Cache image tiles downloaded from the server while the project is open,  saved within the .map project itself in a Cache subfolder in the System Data hierarchy. Also provides greater flexibility with read-only data sources.

Save cached data between sessions

Save the cached data for the next time this project is opened,  within the .map project itself in a Cache subfolder in the System Data hierarchy.  Caution: checking this box can result in very large .map files when the image results of browsing web servers are all saved.  However, having such data cached in the .map is handy for offline browsing of the project.

Tile size

Size of tiles issued by the server.  Always square, so 256 means tiles that are 256 pixels wide by 256 pixels high.  Some servers use 512 pixel tiles.   See the documentation for the server.

Zoom Range

0 is zoomed all the way out and 20 is zoomed all the way in to the closest zoom served by the server.  The closest zoom varies by server and must be entered correctly.   See the documentation for the server.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Application key

A secondary application key or authentication code for those servers that require it.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

Control Sequences

The Custom imageserver dataport understands the following control sequences, as typically used:

 

{x}, {y}

X and Y coordinates of a tile

{zoom}

Zoom level of a tile.  Zoom level is a numeric value similar to an image level but going in the reverse direction: an image level of 0 is the most detailed level, while a zoom level of 0 is the least detailed level supported by the server.   Some URL connection strings specified for servers use the control sequence token {z}.  Replace {z} with {zoom} in the URL to construct a URL to use as the  Source string.

{key}

API key, if entered by the user.

{keyApp}

Application key, if entered by the user.

{lon}, {lat}

Longitude and latitude of the center of the requested tile.

{switch:a,b,c}

Server selector for load balancing: Choices after the semicolon are available alternatives.

Examples

Dozens of examples using the Custom dataport have been published in the JOSM sources.map project that is available on the Release 9 Downloads page on the Manifold website.  Most of the web server data sources in that project are created from strings published in the JOSM sources page at https://josm.openstreetmap.de/wiki/Maps page, from which URL strings may be copied for use in the Source box of the Custom web server data source creation dialog, along with other parameters such as maximum zoom.  

 

Following are some examples of entries for various servers from the JOSM sources page.   These show selected tags and not all information for what sometimes are very long entries.   See the JOSM sources page for complete info.

 

OpenStreetMap using French cartographic style:

 

        <name>OpenStreetMap (French Style)</name>

        <type>tms</type>

        <url>http://{switch:a,b,c}.tile.openstreetmap.fr/osmfr/{zoom}/{x}/{y}.png</url>

        <max-zoom>20</max-zoom>

 

OpenStreetMap using German cartographic style:

 

        <name>OpenStreetMap (German Style)</name>

        <type>tms</type>

        <url>http://{switch:a,b,c,d}.tile.openstreetmap.de/tiles/osmde/{zoom}/{x}/{y}.png</url>

        <max-zoom>18</max-zoom>

 

In both the above cases we can copy the string between the <url> and </url> tags to use as our Source string.  In the case of the German style version we must change the Zoom range value in the New Data Source dialog from [0, 20] to [0, 18]

 

OpenTopoMap

 

        <name>OpenTopoMap</name>

        <type>tms</type>

        <url>http://tile.opentopomap.org/{zoom}/{x}/{y}.png</url>

        <min-zoom>3</min-zoom>

        <max-zoom>17</max-zoom>

 

We can copy the string between the <url> and </url> tags to use as our Source string.  We must change the Zoom range value in the New Data Source dialog from [0, 20] to [3, 17]

 

OpenCycleMap

 

        <name>OpenCycleMap</name>

        <type>tms</type>

        <url>https://{switch:a,b,c}.tile.thunderforest.com/cycle/{zoom}/{x}/{y}.png?apikey=7bd5ed2197cf4da29fa26de0ba6530cc</url>

        <max-zoom>22</max-zoom>

 

An example using an embedded API key, apparently issued by the originator for use by people creating features for OpenStreetMap.  We can copy the string between the <url> and </url> tags to use as our Source string.  We must change the Zoom range value in the New Data Source dialog from [0, 20] to [0, 22]

 

Some of the web servers in the JOSM sources.map project are licensed only for creating features in OpenStreetMap while others are completely unrestricted.  See the comments in the project for links that give complete info.

 

Notes

Problems connecting - Check the Log Window to see what is going on behind the scenes if an attempted connection does not work.     The problem is usually a wrong connection string, failure to provide required credentials such as an key string that authorizes access, wrong choice of protocol (the server uses WMS and the user picks something else), an incredibly slow server, a server that is offline or a server that is wrongly configured and which is not correctly using the protocol it claims to use.  

 

Visit the Manifold community forum and talk out difficulties with other users.  Make sure to post full information on what you are doing, the connection URL you used, all details of how you tried to connect (including all settings in the data source dialog), what happened, and what the Log Window reported.    If other users cannot help you, spending a tech support incident will produce an authoritative analysis of the issue.

 

The very last thing to consider is the possibility that a bug in Manifold is causing the problem.  Bugs happen, of course, but leaping to the conclusion as a first step that a bug is the problem is almost always a mistake.

 

Try the URL in a browser - Checking the URL by launching it in a browser can reveal many problems with the URL or with the web server.  If a URL does not work in a Manifold web server dataport, try exactly the same URL in a browser.  If a browser cannot connect to the URL, the Manifold web server dataport will not be able to connect to it either.  If a browser cannot connect to that URL, that indicates the problem is the URL or the web server.   For example, the web server might be offline.  Or, for example, If the browser connects to a page other than the actual web service endpoint, such as, to a web page that lists various options for web servers, that shows the URL is not a URL for a web server but a URL to some other sort of web page.   

 

Connection problems are often caused by incorrect URLs.  There might be a typographical error in the URL or the URL might not be an endpoint to a functioning server but instead a URL to some other web page.  The server responding to the URL may have geographic restrictions (surprisingly common in a time of geopolitical hysteria) that does not respond to connections from IP addresses that are thought to be in a canceled country.  Trying the URL in a browser will fail in such cases.  Web servers may also have other restrictions, such as only allowing connections from white listed IP addresses, from paying clients, or from those clients that use a special security scheme.  

 

Web servers are often slow or mis-configured -   Some web servers to which we might connect using TMS, WMS, WMTS, WFS or other connection standards can be incredibly unresponsive or, despite allowing a connection, will not allow data to be fetched.   If pressing the Test button results in a successful connection, or if layers available are displayed when we expand the data source, but then when we try to display an advertised layer nothing happens, we can open the Log Window to see what is going on.

 

Here are a few lines of log window information for a happy session, connecting to a WMS server in New Zealand run by the ever-efficient LINZ organization:

 

Web request: (root)::[Data Source] (GetCapabilities) (0.578 sec, 13.0 KB)

Web request: (root)::[Data Source]::[layer-767] (GetMap) (1.255 sec, 47.9 KB)

Render: [Data Source]::[layer-767 Image] (1.561 sec @1.0)

 

In contrast, a few lines of log window information for an unhappy session, attempting to connect to the WMTS server run by the French cartographic service IGN but getting no image for a cadastral parcels layer:

 

Web request: (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) (0.070 sec, 0 B)

*** (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) The remote server returned an error: (403) Forbidden.

Render: [IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b Image] (0.315 sec @1.0)

 

From the above we see an empty cadastral parcels image was rendered because the remote server refused to provide tiles in response to the GetTile request, returning a 403 error Forbidden.    Finding out why a web server refuses service can involve some detective work.  

 

The server might require a special key for access, a key we are trying to use may be out of date, a login and password may be required for some resources, the connection string may have been copied and pasted inaccurately or, of course, in the case of IGN, the server may be refusing to work to show solidarity with striking railroad workers or air traffic controllers.

 

Keep in mind that just because a web server offers tiles via various standards like WMTS or whatever does not mean that it will be fast or even reasonably tolerable.  Some web servers are painfully, incredibly slow and will take many seconds or even minutes per tile.   When images do not appear or fill in with painful slowness a look at the Log Window can show the problem is not our Internet connection or Manifold, but just incredibly slow response from the web server.

 

Some webmasters may insist their web servers are correctly configured even when explicit errors are pointed out.   That can happen even in the case of  national cartographic services as reported, for example, as discussed in various threads in the georeference forum.  

 

User-unfriendly server configurations - Correctly configured web servers may sometimes make user unfriendly configuration choices, for example, only starting to show data when the view is zoomed in to some specific area. That is a perfectly legal configuration, but it may catch users by surprise if they open a web served layer and it is blank, because they have not yet zoomed sufficiently far into a particular region.  If a web server layer is blank, add it to a map with a "known good" background layer, such as Bing Streets, and then zoom into the area of interest that is likely to be covered.  The Geosciences Australia web server, for example, often serves blank layers until one zooms into Australia, and at times far into Australia.

 

Must Have Internet - It seems obvious that to connect to a web server we must have a live, functioning Internet connection.  What might not be obvious is that sometimes we may have only a partial Internet connection, without realizing we do not have a full Internet connection.  A classic example is using a browser, such as Opera, which may ignore failed IPv6 DNS resolution but work OK using IPv4 DNS resolution.  Without IPv6 DNS resolution we can browse Internet using Opera but we might not be able to connect to Microsoft, Google, or some other web servers.   

 

Manifold's web server connections, just like the latest Google Chrome and Microsoft Edge browsers, require full IPv6 DNS resolution.   That normally happens automatically using whatever IPv6 DNS servers are assigned by our Internet service provider, but at times IPv4 DNS continues to work OK while the IPv6 DNS servers assigned by our ISP do not work.  If we are using Opera, and not Edge or some other browser that uses IPv6 DNS resolution, we might not notice that our ISP-provided IPv6 DNS resolution is not functioning.    We might only notice the problem when trying to use an image server or other web server within Manifold and it does not work.

 

If we cannot get a display from a web server, we can check the Manifold Log Window. If the log reports that a connection cannot be made, in Windows network dialogs for our Internet connection  we can try setting the IPv6 DNS servers to some "known good" DNS servers.   A good choice is using Google's public DNS service, using the following addresses for primary and backup DNS server addresses:

 

2001:4860:4860::8888
2001:4860:4860::8844

 

Collections of Web Servers - Check the Product Downloads page for pre-packaged project files in Release 9 / Viewer .map format that Manifold publishes which contain collections of dozens of popular web servers data sources.

 

See Also

Project Pane

 

Web Servers

 

File - Create - New Data Source

 

ArcGIS REST Servers

 

CSV Servers

 

GeoJSON Servers

 

JSON Servers

 

TMS Servers

 

WFS Servers

 

WMTS Servers

 

Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as image servers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.

 

Example: Vector Layers from an ArcGIS REST Feature Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that shows petroleum fields in Kansas, getting the data as a vector drawing layer. Style the layer as if it were local.  ESRI refers to ArcGIS REST servers that provide vector data as feature servers.  

 

Example: Connect to a WFS Server for State Government Data - Gathering our courage, we connect to a WFS server that provides 1200 vector layers, run by the state of Massachusetts.   We open a layer showing airports and then scrape the vector data into our own local storage.  

 

Example: Connect to a WMS Server for National Map Layers - Visit the National Map services web page, copy a URL for a shaded relief layer from USGS, and then use Style to enhance that shaded relief data for combination with other layers and really spectacular effects.

 

Example: Connect to a Custom Server for Cadastral Data - We connect to a custom image server that provides cadastral information originally from the French national cartographic agency, IGN.  We create a map and use the Style pane to re-style the web served image on the fly into a more usable form.

 

Example: Connect to a Custom OpenRailwayMap Server - We connect to a custom server that provides an OpenRailwayMap view of railroads worldwide, showing railway, tram, and subway infrastructure based on OpenStreetMap data.  Our first try at creating a data source does not work.  After consulting the Log Window we try again with a slight adjustment and our second try works.

 

Example: Connect to an OSM Vector Server - We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database.  We then show how to scrape (copy) data from the OpenStreetMap server into local storage.  We extract building footprints from the local copy.

 

Example: Raster Layers from an ArcGIS REST Image Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that provide a raster layer showing a mosaic of aerial photographs near Portland, Oregon.