We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database. We then show how to scrape (copy) data from the OpenStreetMap server into local storage. We extract building footprints from the local copy.
See the video version of this topic in the 5 Minute Tutorial - Connect to an OSM Vector Server and Harvest Data video.
This topic uses a connection
to a web server that is operated by a third party. Web servers
operated by third parties may change their access policies, use different
URLs, go offline, or be very slow. If you have any connection problems
with the server in this topic, review the Notes
at the end of the Web
Servers topic for tips.
It is usually convenient to work with web server layers within a map, so we can always have a "known good" background layer for context.
Web server layers are often very slow as well, so it is especially convenient to be able to use a background layer in a map, like Bing streets, to pan and zoom into our particular area of interest before dragging and dropping a web server layer into the map.
Therefore, many experienced users will begin a project using a new web server layer by first creating a map.
We create a map by right-clicking into a blank portion of the Project pane, and choosing Create - New Map.
For a base layer, we use the default Bing streets background layer, one of the standard Favorites built into Manifold.
Press Create Map.
A new map appears in the Project pane. The system also creates a new Bing Maps Street Map data source for the Bing streets base layer used in the map.
We double-click the Map to open it.
The map opens up zoomed to fit, with the Bing layer showing the entire world.
In Manifold, choose File - Create - New Data Source from the main menu, or right-click in the Project pane and from the menu choose Create - New Data Source.
Choose More... to launch the New Data Source dialog.
Enter OpenStreetMap as the Name. We can enter whatever name we like, but it is wise to specify a name that reminds us of what this data source contains.
Choose Web Server: osm as the Type. The Source box automatically fills with the default URL to connect to the main OpenStreetMap vector layer server.
Make sure to check the Cache data box, which is checked by default. This stores in cache memory everything Manifold downloads from the server, so that subsequent operations, such as panning and zooming, can take the data from cache and not have to download it again.
We leave the other settings at their default values, which usually work.
Press Create Data Source.
A new data source called OpenStreetMap appears in the Project pane.
We can expand the data source to see it contains a drawing called Osm Drawing and the drawing's table, called Osm.
To see what our new OpenStreetMap data source does, we first zoom our map to our area of interest, the part of the city of Chartres, in France, near the great cathedral.
We drag and drop the Osm Drawing layer into the map:
The Osm Drawing layer appears as a tab in the map, with a blue underline indicating it is loading. After about fifteen seconds, more or less, objects begin to appear in the Osm Drawing layer, and after about 30 seconds the layer has filled in and the blue underline in the layer tab disappears, indicating the data for the view has loaded.
The OpenStreetMap data source has sent data to fill the view. If we zoom out or pan to a different location, or if we make the map window larger, we will have to wait for more data to be downloaded to fill in whatever new parts of the scene come into view. If we zoom far out, for example, to the entire world, the OpenStreetMap servers will not send any data, since they assume we have made a mistake in asking for many terabytes of data.
The key strategy for working with OSM servers is therefore to create a map that has a useful background layer, like Bing, that provides orientation, and to use that map to pan and zoom to a view that shows our area of interest. Only then can we add the drawing layer from the OSM server to the map. Or, if we like, we can turn off the OSM drawing layer and then turn it on only when we have the desired view.
Showing the OSM drawing layer only when we are looking at a relatively limited area of interest will keep the number of objects in view down to a number that is small enough for the OpenStreetMap servers to be willing to serve the vector data to us.
Scraping means to grab data that is resident on a web server, downloading a copy of the data onto our own computer system for local use, so we do not need to use the web server. With Manifold that is very easy to do when using OSM servers, the sole obstacle being the time required to download vector data from the relatively slow servers, and not trying to scrape so much data that the OpenStreetMap servers will decide we are asking for too much data.
If we want larger data sets, a much better way to get OSM data is to download an entire data set for a country, as discussed in the PBF .pbf, OSM, O5M and the SQL Example: Extract Airport Runways from an OpenStreetMap PBF topics.
However, for scraping smaller amounts of OpenStreetMap data in a simple way, it can be convenient to use an OSM server. Here is how:
With the focus on the Osm Drawing layer in the map window, Ctrl-click and drag a selection box around the objects to be selected. The illustration above has had magenta color added to better show the selection box, given the very busy background.
That will select all objects that fall within or touch the selection box. The selected objects will be shown in red selection color.
Press Ctrl-C, the universal Windows keyboard shortcut for Copy. Or, in the main menu choose Edit - Copy.
Right-click into the main part of the Project pane, outside of the OpenStreetMap data source, and choose Paste. Or, we can click into the main part of the project to move the focus there, and then press Ctrl-V, the universal Windows keyboard shortcut for Paste, or we could click into the main part of the project and then from the main menu we could choose Edit - Paste.
A new table called Osm appears. This table contains the data for the selected objects that we copied from the Osm Drawing layer. We have just scraped the data for those objects from the OpenStreetMap server and have pasted it into our local project.
To use that new table, we will create a drawing for it. Right-click on the new Osm table in the project, and choose Create - New Drawing.
The coordinate system appears in red because we have not yet specified the coordinate system that the OSM geometry field uses. We click the coordinate picker button and choose Latitude / Longitude, a one-click favorite in the menu that appears.
If we did not know that OSM data uses Latitude / Longitude projection, with the focus on the Osm Drawing layer in the map we could have consulted the Info pane, which would have reported the projection in use was Latitude / Longitude.
Press Create Drawing.
A new drawing called Osm Drawing appears in the project pane. We drag and drop that new drawing into the map window.
In the illustration above, we have turned off the Osm Drawing : OpenStreetMap layer, to better see the new layer we have added. As expected, the new layer is all of the objects that were selected in the Osm Drawing : OpenStreetMap layer, which we copied and then pasted into the new table, and thus into the new drawing which we created from the new table.
It may be puzzling that we can use the same name, Osm Drawing for two different components in the same project. We can do that because the full name for the original drawing is Osm Drawing : OpenStreetMap, as it is inside the OpenStreetMap data source. We can use the same names, so long as they are in different data sources.
If we pop open the Osm table, we can see the many fields brought in from OpenStreetMap. Much of the attribute information that interests us, which we would use to untangle and separate the various lines and points, is in the tags attribute field.
We use the Layers pane to turn off all the fields except the geometry field and the tags field, and we scroll down to near the bottom of the table. We can see that OpenStreetMap data contains a staggering amount of information. For example, in this case there are entries for individual benches (as point objects), specifying whether those benches have backrests or not, the color and material of the bench, and how many seats in the bench.
The main issue with OpenStreetMap, when working directly with data from OpenStreetMap, is not so much getting the data as it is extracting useful subsets of data from the staggering mass of detail.
Lucky for us, Manifold has really super tools for working with data and attributes. We will show how easy those are to use by extracting the building footprint lines from the data we have scraped.
With the focus on the Osm table window, in the Select pane we choose the tags field, and then we double-click the Search template to launch it. We could also do exactly the same thing with the focus on Osm Drawing layer in the open map window, choosing the Osm Drawing layer and the tags field.
In the Search template, we choose text as the Use option (text is the default), and for Condition we choose contains.
In the Value box we enter "building": and we leave other options at their default settings. We will select all records where the tags field contains the text "building": .
Press Select.
That selects all all records where the tags field contains the text "building":, and shows them in red selection color.
In the map window, we can see that all of the building footprint lines have been selected, and are shown in red selection color.
With the focus either on the table window, or on the Osm Drawing layer in the map window, press Ctrl-C or choose Edit - Copy in the main menu to Copy those records.
Next, in the Project pane right-click into the main part of the project and choose Paste.
That creates a new table called Osm 2. We change the name of that table to Buildings and then using the same procedure we did earlier, we create a drawing from that table called Buildings Drawing, remembering to specify Latitude / Longitude as the coordinate system in the New Drawing dialog.
Drag and drop the new Buildings Drawing layer into the map, and we get a display like that above. In the illustration above we have turned off the Osm Drawing layer.
If we take a close look at the illustration, we see that OpenStreetMap data does not fully agree with Bing's map of Chartres. There are cases where streets that Bing shows run through the middle of OSM building footprints.
We take a moment to create an OpenStreetMaps Base imageserver layer and we add it to the map in a layer above Bing, turning off Bing. It seems that the OSM footprints agree with the OSM imageserver.
We zoom into the view, just to the South of the great cathedral, to take a closer look, with Le Café Serpente at the center of the view. We can see the agreement is very close, as expected, since both the OSM imageserver and the OSM building footprints came from the same database.
Turning the OSM imageserver layer off, and turning Bing back on, we can see that Bing strongly disagrees, with Rue au Lait running right through the Café Serpente building.
For a tie-breaker, to see if OSM or Bing is right, we can use a Google satellite layer.
We add a Google Maps Satellite layer, and we use the Style pane to draw the Buildings Drawing lines in bright green. We can see that the OSM building footprints align well with the Google Maps Satellite imagery.
To see what the place looks like in real life, we can fire up Google Maps in our browser, and take a look at Le Café Serpente in Google Street View, shown above. The spectacular cathedral is just visible at the left edge of the view in the illustration above.
For more on Chartres Cathedral, and to learn how it was nearly destroyed but for an American who gave his life on the mission he undertook to save it, see the Example: Locations topic.
OSM Data is Big - OSM data is way too large to simply scrape by copying the drawing or table within the OpenStreetMap data source we created and then pasting into our drawing. Surprisingly, Manifold can handle that on a desktop machine if we have a few spare terabytes of free storage, but even if the OpenStreetMap organization was willing to serve that much data through the web, it would take inconveniently long to download.
A much better idea is to visit the geofabrik.de site at https://download.geofabrik.de/europe.html and download the OpenStreetMap PBF for France. That is only a 3.7 GB PBF, which will download reasonably quickly if we have a fast Internet connection, and which is straightforward to import into Manifold.
See the SQL Example: Extract Airport Runways from an OpenStreetMap PBF for an example.
5 Minute Tutorial - Connect to an OSM Vector Server and Harvest Data
File - Create - New Data Source
Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as image servers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.
Example: Vector Layers from an ArcGIS REST Feature Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that shows petroleum fields in Kansas, getting the data as a vector drawing layer. Style the layer as if it were local. ESRI refers to ArcGIS REST servers that provide vector data as feature servers.
Example: Connect to a WFS Server for State Government Data - Gathering our courage, we connect to a WFS server that provides 1200 vector layers, run by the state of Massachusetts. We open a layer showing airports and then scrape the vector data into our own local storage.
Example: Connect to a WMS Server for National Map Layers - Visit the National Map services web page, copy a URL for a shaded relief layer from USGS, and then use Style to enhance that shaded relief data for combination with other layers and really spectacular effects.
Example: Connect to a Custom Server for Cadastral Data - We connect to a custom image server that provides cadastral information originally from the French national cartographic agency, IGN. We create a map and use the Style pane to re-style the web served image on the fly into a more usable form.
Example: Connect to a Custom OpenRailwayMap Server - We connect to a custom server that provides an OpenRailwayMap view of railroads worldwide, showing railway, tram, and subway infrastructure based on OpenStreetMap data. Our first try at creating a data source does not work. After consulting the Log Window we try again with a slight adjustment and our second try works.
SQL Example: Extract Airport Runways from an OpenStreetMap PBF - We write a simple SQL query using INNER JOIN to extract runway lines from an OpenStreetMap PBF of Cyprus, and to save those lines to a new drawing and table.
Example: Raster Layers from an ArcGIS REST Image Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that provide a raster layer showing a mosaic of aerial photographs near Portland, Oregon.