ArcGIS REST servers are web servers that provide raster or vector layers using REST (representational state transfer) architecture and ESRI ArcGIS protocols. ESRI uses the name feature server to refer to ArcGIS REST servers that provide vector data. We will use feature server the same way in this example topic. We connect to an ESRI feature server as a data source, allowing us to pull vector layers from that server for display. We copy vector data ("scraping") from the ESRI server into our local project to use off line.
This topic uses a connection
to a web server that is operated by a third party. Web servers
operated by third parties may change their access policies, use different
URLs, go offline, or be very slow. If you have any connection problems
with the server in this topic, review the Notes
at the end of the Web
Servers topic for tips.
For an example showing a connection to an ArcGIS REST server for raster data, see the Example: Raster Layers from an ArcGIS REST Image Server topic.
In our browser (Microsoft's Bing browser is illustrated) we can visit the ESRI page at https://sampleserver3.arcgisonline.com/ArcGIS/rest/services/ for links to a variety of example ArcGIS REST feature (vector) servers operated by ESRI.
Double-click on the Petoleum link to visit that page.
The Petroleum page lists a variety of servers. The FeatureServer serves vector layers, while the Mapserver entries serve raster tile layers. We want the feature server.
Right-click onto the Petroleum/KSFields feature server link, and choose Copy link from the menu. The URL we copy is:
https://sampleserver3.arcgisonline.com/ArcGIS/rest/services/Petroleum/KSFields/FeatureServer
The URL above is all on one line. If the browser displaying this topic wraps it into two lines, when using the URL above make sure to copy and paste it as a single line.
In Manifold, choose File - Create - New Data Source from the main menu, or right-click in the Project pane and from the menu choose Create - New Data Source.
Choose More... to launch the New Data Source dialog.
Enter KSFields Petroleum as the Name. We can enter whatever name we like, but it is wise to specify a name that reminds us of what this data source contains.
Choose Web Server: arcgisrest as the Type.
In the Source box we paste the URL connection string we copied earlier:
https://sampleserver3.arcgisonline.com/ArcGIS/rest/services/Petroleum/KSFields/FeatureServer
Make sure to check the Cache data box, which is checked by default. This stores in cache memory everything Manifold downloads from the server, so that subsequent operations, such as panning and zooming, can take the data from cache and not have to download it again.
ArcGIS REST feature servers can be incredibly slow, so once we suffer the pain of waiting for data to be sent by the server we do not want to wait any more to re-fetch the data from the server.
Press Create Data Source.
A new data source called KSFields Petroleum appears in the Project pane.
We can expand the data source to see it contains a drawing, the drawing's table, and a comments component.
We could use the drawing from the data source as is, but it would be more convenient to create a map in the local part of the project, and then to create a version of the drawing in the project, which we can Style.
For the difference between parts of the project that are resident in remote data sources and those parts of the project that are resident locally, see the Portals to Other Worlds discussion in the Project Pane topic.
We create a map by right-clicking into the local part of the project and choosing Create - New Map. We create a map that has a Bing streets background layer, otherwise having no layers.
To give us flexibility styling the drawing that is served by the ESRI feature server, we will make a local copy of the drawing, but not the table.
We right-click the Current Kansas Field Production Drawing and choose Copy. Next, we right-click down in the local part of the project, in the blank area below the Map component, and we choose Paste.
That creates a local copy of the drawing, also called Current Kansas Field Production Drawing, which we can use as it if were the original drawing in the data source. The drawing we have just created takes its data from the table that is within the data source, but because it is a local structure under our full control in the local project, we can style it as we like.
We double-click the Map to open it. The map opens showing the single layer it contains, a Bing streets layer.
We know the drawing served by the ESRI feature server shows petroleum fields in Kansas, so we pan and zoom to show Kansas. We are not required to do this, but because we know that ESRI feature servers can be very slow, we feel we may as well pan and zoom to where the data will (eventually) show up.
If we are following along in this example and repeating it, now is the time to brace ourselves for a very slow response from the server. The ESRI server is incredibly slow to send vector data. When we drop the drawing from the KSFields Petroleum data source into the map, it will seem to take forever to populate. This is not Manifold being slow: it is a slow server sending data using a slow protocol.
Drag the Current Kansas Field Production Drawing (the copy we created, not the original within the data source) and drop it into the map. We go get a cup of coffee or a stiff drink.
While the drawing slowly populates (minutes of time!) a blue bar will underline the drawing's tab to show it is still pulling data from the server. As data comes in, various vector shapes will appear in the drawing.
Eventually, the entire drawing will be received from the server, and the blue underline will disappear. From here on in, so long as we checked the Cache data box (it is checked by default) when creating the data source, performance will be much better. We will be able to pan and zoom and open the drawing's table without waiting for data to be downloaded from the server, since Manifold will fetch needed data from local cache.
We can pop open the drawing's table, double-clicking the Current Kansas Field Production Layer table within the KSFields Petroleum data source.
The table pops open with a gray background, since it is a read-only table. Vector data served by ArcGIS REST servers can be very slow to download, but once we get it we often get the benefit of attribute data, something we do not get from raster tile servers such as WMS servers.
Consulting the Layers pane, we can see the many attribute fields provided with this web-served drawing.
We can Style the drawing using those attributes.
For example, in the Style pane we can style the fill color for areas using the activeprod field to show which fields have gas production, oil production, or both oil and gas production.
Press Update Style.
The drawing layer in the map immediately updates to show the thematic format we used. We have also reduced the thickness of border lines for areas to .5 point.
We have accomplished our task, connecting to a remote ESRI feature server and using a vector layer from that server in a project. We continue with this tutorial example to show other moves that are very useful in real life, done the way an experienced Manifold user might do them.
With the project as we currently have it arranged, the KSFields Petroleum data source is a portal into the ESRI ArcGIS REST feature server sitting somewhere in North America. We have copied the drawing into a local component within the map project, but a drawing is just a window into data stored in a table somewhere, and in this case, the Current Kansas Field Production Drawing we have in the local part of the map takes its data from a table that is stored in the distant ESRI ArcGIS REST feature server.
We can see that by right-clicking on the local Current Kansas Field Production Drawing and choosing Properties.
We can see from the Table entry above that the table the drawing gets data from is the Current Kansas Field Production Layer table within the KSFields Petroleum data source.
On the one hand, that is a good thing, because one reason people create feature servers is to publish data that changes. When we connect to a feature server, if the data on the server has changed, the table data and thus any drawing created from that table data will be fresh and up to date. That is true as well for any local drawings we create that take their data from the table the server provides. If the data in the table changes, the drawings created from that table will also change.
On the other hand, we might want a fully local copy of the data set that does not depend in an way on a remote server. For example, we may be working with data that we do not expect to change, where grabbing a snapshot of what is on the server for local use is perfectly OK. We might prefer to work with local data so we can continue working without an Internet connection, or simply because we cannot tolerate the incredible slowness of data provided by remote servers.
One nice thing about feature servers is that it is easy to copy data from the feature server into the local project, as easy as copy and pasting files in Windows Explorer.
Ctrl-click both the drawing and the table in the data source to highlight them. Right-click onto the highlighted items and choose Copy.
Next, Right-click into the main part of the project and choose Paste.
Done. A new drawing and its table appear in the Project pane. Since we already have a drawing called Current Kansas Field Production Drawing in the local project, Manifold will paste the additional drawing using the name Current Kansas Field Production Drawing 2.
The copy and paste happens very fast because the data from the ESRI server is already in local cache. There is no need to download it again.
If we drag and drop that new drawing into the map, we see that it is, indeed, the same drawing we have been using from the ArcGIS REST feature server (we have used Style to reduce the thickness of border lines for areas to .5 point).
But now the drawing takes its data from a purely local table, as we can see by opening the Properties dialog for the drawing.
Unless we want to stay connected to the KSFields Petroleum data source, we do not need it anymore, and we could delete it if we want.
We Ctrl-click the KSFields Petroleum data source to highlight it, and then we press the Delete button in the Project pane toolbar.
Deleting the data source leaves the Current Kansas Field Production Drawing layer in the map without a source of data to display, because we have just deleted the data source and the table in that data source that drawing was using. We can delete that drawing, or, if we do not want to waste the nice thematic format we created in it, we can point the Current Kansas Field Production Drawing to using the local Current Kansas Field Production Layer table.
Right-click onto the Current Kansas Field Production Drawing and choose Properties.
In the Properties dialog, we double-click into the value cell for the Table property to start editing the text, we highlight the reference to the now-deleted data source name, and we delete it.
Press OK. That updates the properties for the Current Kansas Field Production Drawing so it takes its data from the local table, and not from a now-deleted data source table.
Like magic, the Current Kansas Field Production Drawing layer comes back to life, since now the drawing in that layer has a table that it can pull data from. The table is the completely local Current Kansas Field Production Layer table, which has no connection to the original ArcGIS REST feature server.
By copying and pasting that table from the feature server data source into the project, we have "scraped" the data from the ESRI server and now have our own copy in the local project. That copy and paste went very fast because the data from the ESRI server was already in local cache.
File - Create - New Data Source
Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as image servers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.
Example: Connect to a WFS Server for State Government Data - Gathering our courage, we connect to a WFS server that provides 1200 vector layers, run by the state of Massachusetts. We open a layer showing airports and then scrape the vector data into our own local storage.
Example: Connect to a WMS Server for National Map Layers - Visit the National Map services web page, copy a URL for a shaded relief layer from USGS, and then use Style to enhance that shaded relief data for combination with other layers and really spectacular effects.
Example: Connect to a Custom Server for Cadastral Data - We connect to a custom image server that provides cadastral information originally from the French national cartographic agency, IGN. We create a map and use the Style pane to re-style the web served image on the fly into a more usable form.
Example: Connect to a Custom OpenRailwayMap Server - We connect to a custom server that provides an OpenRailwayMap view of railroads worldwide, showing railway, tram, and subway infrastructure based on OpenStreetMap data. Our first try at creating a data source does not work. After consulting the Log Window we try again with a slight adjustment and our second try works.
Example: Connect to an OSM Vector Server - We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database. We then show how to scrape (copy) data from the OpenStreetMap server into local storage. We extract building footprints from the local copy.
Example: Raster Layers from an ArcGIS REST Image Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that provide a raster layer showing a mosaic of aerial photographs near Portland, Oregon.