Gathering our courage, we connect to a WFS server that provides 1200 vector layers, run by the state of Massachusetts. We open a layer showing airports and then scrape the vector data into our own local storage. We gather our courage, because we know that WFS servers can be slow and unresponsive, especially while we wait for all 1200 layers to be reported as available.
This topic uses a connection
to a web server that is operated by a third party. Web servers
operated by third parties may change their access policies, use different
URLs, go offline, or be very slow. If you have any connection problems
with the server in this topic, review the Notes
at the end of the Web
Servers topic for tips.
WFS servers provide vector data, that is, drawings with attribute tables. While it is nice to get vector data from a server, perhaps even vector data that is continuously updated, the relatively slow interactive performance of WFS servers can be discouraging to Manifold users who become used to everything happening very fast using local data. Experienced GIS users will often browse WFS servers to see what data an agency makes available, and will then put their efforts into downloading the data from an FTP or other web site for local use.
We visit the Mass GIS (Massachusetts GIS) state wiki page at https://wiki.state.ma.us/display/massgis/GeoServer+-+WFS+-+GetCapabilities to find a GetCapabilities URL. That is usually a good URL to use to connect to a WFS server. Other web sites that announce data layers that are available may have a WFS button with a link to a URL to get that layer.
In the case of Mass GIS, the GetCapabilities URL is:
http://giswebservices.massgis.state.ma.us/geoserver/wfs?request=GetCapabilities&service=WFS&version=1.0.0
We copy that URL from the website.
In Manifold, choose File - Create - New Data Source from the main menu, or right-click in the Project pane and from the menu choose Create - New Data Source.
Choose More... to launch the New Data Source dialog.
Enter Mass GIS as the Name. We can enter whatever name we like, but it is wise to specify a name that reminds us of what this data source contains.
Choose Web Server: wfs as the Type.
In the Source box we paste the URL connection string we copied earlier:
http://giswebservices.massgis.state.ma.us/geoserver/wfs?request=GetCapabilities&service=WFS&version=1.0.0
Make sure to check the Cache data box, which is checked by default. This stores in cache memory everything Manifold downloads from the server, so that subsequent operations, such as panning and zooming, can take the data from cache and not have to download it again. That makes a big difference for interactive use. Once we wait for data to be sent by the server we do not want to wait any more to re-fetch the data from the server.
Press Create Data Source.
A new data source called Mass GIS appears in the Project pane.
We can expand the data source to see it contains very many layers. The GetCapabilities URL we used asks the server to report all layers it can provide via WFS. In the case of Mass GIS, that is about 1200 layers, and it takes the server tens of minutes to report them all. As the server reports them, the list within the data source will fill in. When all layers are reported, the small black triangle, indicating "running," on the yellow cylinder icon in the data source will disappear.
We can work, somewhat, with layers that have already filled in while waiting for the server to report them all, but response will be painfully slow while the server is also occupied with reporting all layers available. Once those are all reported, the server will be much more responsive.
We can open the Airports folder that appears, and double-click the drawing within that folder to open it. There are only 42 airports in Massachusetts, and in this data set they are represented by points, so we can be brave and risk taking the time that the server takes to send geometry and attributes for 42 points.
A few tens of seconds, and the drawing fills in with points. After the server finishes reporting all layers, a layer with only 42 points will appear in a few seconds. Once Manifold cache kicks in, the layer will appear instantly the next time we open the drawing or pan and zoom in it.
If we like, we can double-click the drawing's table to open it, to see attributes.
One nice thing about WFS, is that so long as we checked the Cache data box when we created the WFS data source, we can select within tables and drawings served by WFS servers. For example, in the illustration above we have ctrl-clicked the record for LOGAN INTERNATIONAL AIRPORT in Boston.
The point for Logan airport is immediately selected in the drawing window, as well.
Scraping means to grab data that is resident on a web server, downloading a copy of the data onto our own computer system for local use, so we do not need to use the web server. With Manifold that is very easy to do when using WFS servers, the sole obstacle being the time required to download data sets from slower servers.
In the Project pane, we expand the Mass GIS data source and navigate our way down to the Airports folder, where we ctrl-click both the drawing and the drawing's table, to highlight both. We must highlight both to copy both. We then right-click onto either highlighted item and choose Copy from the context menu.
Next, we close up the Mass GIS data source (so that a list of 1200 layers does not get in our way) and we right-click into a blank portion of the project outside the Mass GIS data source. We choose Paste from the context menu.
Another way to copy and paste is to highlight what we want to copy, press Ctrl-C as the standard Windows keyboard shortcut for Copy, and then click into a blank part of the project to move the focus there, and to press Ctrl-V, the standard Windows keyboard shortcut for Paste. If we like to use menus, we can choose Edit - Copy and Edit - Paste in the main menu, or if we prefer buttons we can use the Copy and Paste buttons in the Project pane toolbar.
However we choose to Copy and Paste, a copy of the table and drawing appear in the Project pane, using the same long names that Mass GIS uses. If we prefer shorter names (we do) we can right-click the drawing and choose Rename related to rename the drawing and the table to something shorter, like Airports. That we do, as seen above.
We now have the airport data in our local project. The Airports drawing and its corresponding table are completely independent of the WFS server. Scraping the data is as simple as copy and paste.
We can open the Airports drawing to see that it does, indeed, contain all the points that the original in the WFS server contained.
We likewise can open the table to see that all the attribute data has been copied as well.
So far, we have been looking at the Airports layer using default, gray formatting, with no background layer to provide context. We can provide a better view by creating a map.
In the Project pane, right-click on the Airports drawing and choose Create - New Map.
In the New Map dialog we use the default name Map and choose Bing Maps Street Map as a base layer from the pull down list of Favorite image servers.
The Airports drawing uses the same local Massachusetts coordinate system as the original layer within the Mass GIS WFS server. Creating a map allows us to use whatever coordinate system we want for the map, and the Airports drawing will be reprojected on the fly into the map's coordinate system. We use the default Pseudo-Mercator coordinate system, since that is what Bing uses.
The dialog creates a new Map for us, and it also creates a new Bing streets image server data source to provide the background layer we chose.
When we double-click open the Map, we see the Airports layer using default gray formatting, showing airports in Massachusetts in a georeferenced layer above the Bing background layer.
While it is nice to see that Manifold is reprojecting the Airports layer on the fly to match the projection used by the map, and it is also nice to see everything happening instantly, since we now are working with local data, the map above has a poor choice of symbology and is difficult to read.
We can fix that by styling the points and by using a better background, one that gives more contrast than the Bing streets layer.
Much better. We have styled the Airports layer to use thematic formats on the servicelevel attribute for size, symbol, and point fill color, to show larger points for primary commercial service airports, which are also colored green and use diamond symbols. General aviation airports are smaller blue circles, with reliever airports of various types in purple and orange-yellow.
For a background we use the Canvas Dark web server from the ArcGIS_REST_Servers.map project that may be downloaded from the Product Downloads page. We open that project in a different Manifold session, copy the data source, and then paste it into our current project session.
Good news and bad - Mass GIS deserves praise for putting 1200 layers up on their GeoServer for people who want to work with GIS data the state government has available. Unfortunately, WFS as a technology means that it may not be very realistic to work with larger layers so long as they stay on the WFS server. Waiting for a million roads to refresh from a really large layer can be a real test of patience, even if we try to work only in zoomed in views where relatively fewer roads need to be fetched. It is possible to scrape such layers by doing a copy and paste, but in the case of a slow WFS server that might be best done overnight for a larger layer. A much better idea is to try to find an FTP site that allows download of the data set in a few minutes. We can then import into Manifold and do in seconds locally what takes uncomfortably long with the WFS server.
File - Create - New Data Source
Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as image servers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.
Example: Vector Layers from an ArcGIS REST Feature Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that shows petroleum fields in Kansas, getting the data as a vector drawing layer. Style the layer as if it were local. ESRI refers to ArcGIS REST servers that provide vector data as feature servers.
Example: Connect to a WMS Server for National Map Layers - Visit the National Map services web page, copy a URL for a shaded relief layer from USGS, and then use Style to enhance that shaded relief data for combination with other layers and really spectacular effects.
Example: Connect to a Custom Server for Cadastral Data - We connect to a custom image server that provides cadastral information originally from the French national cartographic agency, IGN. We create a map and use the Style pane to re-style the web served image on the fly into a more usable form.
Example: Connect to a Custom OpenRailwayMap Server - We connect to a custom server that provides an OpenRailwayMap view of railroads worldwide, showing railway, tram, and subway infrastructure based on OpenStreetMap data. Our first try at creating a data source does not work. After consulting the Log Window we try again with a slight adjustment and our second try works.
Example: Connect to an OSM Vector Server - We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database. We then show how to scrape (copy) data from the OpenStreetMap server into local storage. We extract building footprints from the local copy.
Example: Raster Layers from an ArcGIS REST Image Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that provide a raster layer showing a mosaic of aerial photographs near Portland, Oregon.