Importing and Linking

Most often the data we will want to use in a Manifold project originates in some source outside of Manifold  We often have a choice whether to bring that data into the Manifold project by importing the data into the project, or working with the data left in the original source by linking that data into the project.  Manifold can read virtually every format and data source in existence.   See the Big List of Formats and Data Sources topic for the huge list.   

 

Manifold users usually import data into a Manifold project, to run super-fast without the limitations of older formats.  Older formats are great for interchange, but they cause slow downs and reduced capabilities when used for operational storage.  Of course, Manifold also can link to external data in the original format if we want.  

 

A big difference between Manifold and older GIS:  Manifold itself is a super-fast database system that can store huge data inside the Manifold project.  Older GIS packages like Arc or Q do not have an internal database:  whenever they add vector or raster layers to a project those layers are linked, with the data stored outside the project in the original file.

 

Unlike older GIS packages, with Manifold we can import data for storage within Manifold.  Importing a vector layer from a shapefile with File - Import copies the data from the shapefile into the Manifold project, where it is stored inside Manifold's internal database.  Once the data is in Manifold there is no connection back to the original shapefile.  If we want to save vector data from Manifold into a shapefile, we use File - Export to export the data into a shapefile.

 

In Manifold we also can use File - Link to leave data in the original source format if we want, and work with it in-place the way older GIS packages do, but faster and better.   Manifold can work faster when linking to formats like an ESRI geodatabase or a PostgreSQL/PostGIS database because Manifold itself is a fast, parallel DBMS.  Using Manifold's internal DBMS for caching and other performance enhancements, Manifold can operate a connection to an external database faster than slower, single-threaded clients like ArcGIS Pro or QGIS.

 

For simultaneous, multiuser read-only sharing of spatial data, use Manifold Server.   Manifold Server is a high performance, parallel, spatial database server that is built into Manifold Release 9 Universal edition.  It allows many Manifold users to share data in a .map file, or in a constellation of nested .map files, that is published through Server.

Importing

Importing a file brings data from the file into the Manifold project.   No ties to the originating file remain and the data within the project, such as a drawing or image or table, is a purely Manifold component with all Manifold facilities available.

 

To import a file:

 

  1. In the main menu choose File - Import.

  2. In the Import dialog, navigate to the folder containing the desired file.

  3. Double-click the file to be imported.

 

In most cases Manifold will automatically determine the correct file type to use for the importing dataport based on the "three letter" file extension.  Sometimes when the file type cannot be determined we may have to choose the specific file type from the long list available, click on the file to choose it, and then click Open.

Linking

Linking a file leaves data in storage within the file.   It only appears to be within the Manifold project.  Manifold does a good job of giving the appearance that such linked data can be used as if it were a true Manifold component.  However, because external file formats generally are far more limited than Manifold native storage, not all Manifold facilities might be available with linked data.    For example, some linked files may only be able to store vectors and not rasters, and vice versa.  In contrast, a Manifold table can store anything.

 

To link a file:

 

  1. In the main menu choose File - Link.

  2. In the Link dialog, navigate to the folder containing the desired file.

  3. Double-click the file to be linked.

 

In most cases Manifold will automatically determine the correct file type to use for the linking dataport based on the "three letter" file extension.  Sometimes when the file type cannot be determined we may have to choose the specific file type from the long list available, click on the file to choose it, and then click Open.

 

Using File - Link is a short cut way to create a new data source using a slightly simpler dialog than the New Data Source dialog.   File - Link automatically uses popular options, such as caching data for the duration of the Manifold session.   That is a good option because having a local cache within Manifold can dramatically speed up working with some formats,  such as shapefiles, when linking.  However, having caching turned on will result in a slow File - Link process with large files because Manifold in effect must import the entire data set into local cache before proceeding.   To get a choice of whether local cache is used or not, use File - Create - New Data Source.

 

File - Create - New Data Source is a more sophisticated way of linking data.  File - Create - New Data Source allows us to choose using cache or not, and the File - Create - New Data Source dialog is the way we link data from databases, where optional settings such as login or password for the database might be required.

 

To link a file or a database as a new data source:

 

  1. In the main menu choose File - Create - New Data Source.

  2. Choose More... to launch the New Data Source dialog, if the file or database desired is not already one of the Favorites listed.

  3. In the New Data Source dialog provide a Name for the new data source if the default is not desired.

  4. Choose the Type of data source from the long list available.

  5. Choose the sub-type of data source when those are available.

  6. The Source box will be automatically loaded for some data sources, for example, connection URLs to Bing or Google servers.   For other sources, use the [...] browse button to navigate to the file or data source connection as required.

  7. Set option boxes that appear as desired, for example, using or not using cache, or checking Open as read-only if the data source should be kept read-only.

  8. Press Create Data Source.

 

See the File - Create - New Data Source topic for details.

 

To import data from a linked data source:

 

  1. Open the data source in the Project pane, usually by clicking the + box next to the cylinder icon that indicates a data source.

  2. Drill down within the data source hierarchy to the component desired.

  3. Ctrl-click desired components to highlight them.

  4. Press the Copy icon in the Project pane toolbar, or press Ctrl-C.

  5. Click anywhere in the main part of the Project pane, that is, outside of the data source hierarchy.

  6. Press the Paste icon in the Project pane toolbar, or press Ctrl-V.

 

The above makes a local copy of the data within the project.  It is the same as importing the desired components, as opposed to linking them and leaving them in storage within the external data source.   When copying components such as drawings and images, do not forget to copy both the drawing and the table from which it takes data.  

Drag and Drop to Add Data

Many users prefer to add data to a project by using drag and drop from Windows Explorer.

 

 

 

 

Besides dragging and dropping files and other data sources, you can also add data from Manifold project files in .map or .mxb format as follows:

 

 

 

 

 

Drop data where you want it to go in the project.  If you have folders in the project, dragging a GeoTiff file from a Windows Explorer window and dropping it onto that folder in the Project pane will create the resulting imported image within that folder.   Likewise, if you have linked a database into a project, dropping the dragged GeoTiff file is a command to create the imported image within that database.  If the database connection is read only, or if the database is unable to create an image for the data, the drag and drop will fail.    Drag and drop files into parts of the project where you have full write capability, for example, in the main part of the project outside any data sources that have been linked in.

 

Data sources within a Manifold project are portals to different worlds, that is, to storage outside of the current project file.   Manifold therefore takes care to avoid situations where people casually intending to drag and drop files or other data into a project do not accidentally drop (and thus load) such data into, say, their organization's Oracle database.   Dropping files into a data source pops open a confirmation dialog by default.  The requirement to confirm a drop into a data source can be turned off or back on in the Tools - Options dialog.

 

Data sources are connected/opened on first use, with the database cylinder changing from gray to yellow to indicate the connection.   Dropping files into a data source that has not been opened yet or that is a read-only data source will pop open an error dialog.

 

See the 10 Minute Tutorial - Drag and Drop video for tips on dragging and dropping, including very convenient moves in Windows Explorer that allow dragging and dropping many files at once, even from different folders.

"Unlinking"

Suppose we have linked a file and now we would like to "unlink" it, that is, to import it?  That is easy to do: Copy the desired components in the data source and then Paste them into the main part of the drawing.    "Unlinking" is importing data from within a linked data source.

 

To import data from a linked data source:

 

  1. Open the data source in the Project pane, usually by clicking the + box next to the cylinder icon that indicates a data source.

  2. Drill down within the data source hierarchy to the component desired.

  3. Ctrl-click desired components to highlight them.

  4. Press the Copy icon in the Project pane toolbar, or press Ctrl-C.

  5. Click anywhere in the main part of the Project pane, that is, outside of the data source hierarchy.

  6. Press the Paste icon in the Project pane toolbar, or press Ctrl-V.

 

The above makes a local copy of the data within the project.  It is the same as importing the desired components, as opposed to linking them and leaving them in storage within the external data source.   When copying components such as drawings and images, do not forget to copy both the drawing and the table from which it takes data.  

 

Once we import data from a linked data source, if we no longer want the linked data source in our project, we can delete it.

Import:  Save Data within the .Map Project

Importing data from a file copies data from the originating file format and stores it within the project as a local component, leaving the original file unchanged and preserving no connection to the originating file.   Until the project is saved the data is maintained in a temporary Manifold format file.  When the project is saved to a .map project file the data will be stored in that .map file.   This is an option we do not have in older GIS packages like Arc or Q: it is a key advantage of Manifold to be able to import into superfast Manifold storage.

 

 

Importing large files can take a long time because the imported data will be analyzed and stored in special, pre-computed data structures within the Manifold file that allow subsequent reads and writes to be very fast.  It pays to be patient with such imports as once the data is imported and stored within a Manifold project file access to that data will usually be far faster than it was in the original format.    Once imported the data will open instantly thereafter..

 

Importing data will create tables and other components within the project.   Depending on the data that is imported, importing will create components within the project to display the data in appropriate form.   For example, vector data imported from typical GIS file formats will create one or more tables for the data as well as one or more drawings that display that data in visual form.   Importing data brings it into the project within Manifold infrastructure under full Manifold management for maximum performance.     See comments in the How to Edit a Single File topic.

 

Importing data into a project breaks all connections between whatever it was we imported and the former format.   The Manifold dataport which handles the import will automatically do all necessary conversions and will build all necessary Manifold infrastructure so whatever we import can thereafter be a genuine Manifold thing.   For example, an image imported into Manifold is no longer a JPEG, TIFF or PNG image.  Instead it has become  a Manifold image.   

 

Beginners who say "Oh, I'm working on this JPEG image in Manifold and need to change the style..." are making a classic novice error: an image in Manifold is not a JPEG image.  It is a Manifold image and a Manifold image only.   Whatever we do with it has nothing at all to do with JPEG.  Some day we may want to export it as a JPEG, and if we decide to do that the Manifold dataport doing the export will take the Manifold image and convert it into whatever form is required to save as a JPEG.   

 

Dataports - Dataports are Manifold modules that interact with data sources to provide meaning and structure and access to the data in those data sources in different ways.   A simple dataport knows how to import or to link to data in a specific format.   A more complex dataport knows how to interpret ESRI-arranged data within an Oracle database as an Esri enterprise geodatabase data source.  

Link: Leave Data in the Source File or Database

Linking data from a file establishes at least a read connection and if possible a read/write connection to the originating file format.   This is what older packages like Arc or Q do whenever a layer is "added" - the data stays outside in the original file.   We can do that in Manifold too, if we want.

 

 

Linked components such as tables, drawings or images will appear within the project pane hierarchy and, for fully read/write connections, can be used as seamlessly as if they were stored within the project, but all linked data remains stored in the original file and is not copied or otherwise moved into storage within Manifold.   If the data connection to the linked file is read/write we can edit the data, within whatever limits are imposed by the file format. If the data connection is read-only, we will not be able to edit the data.    

 

Caution: Editing data in a linked file (other than .map project files) that is writable usually will immediately change the data in that file.

 

Saving a project will also save all of the links for linked components in the project.   Linked components take no storage space in a saved .map project file because all that is saved in the project file is a link to the file holding the data.   We should keep that mind when moving the .map project file.   If we send a copy of the .map file to a friend we are only sending whatever data is in the file in local components.   Unless our friend also has a copy of the files that are linked into the project when he or she opens the project file in Manifold the system will not be able to find the files that are supposed to be linked into the project because they will not exist on our friend's machine.

 

 Linking a file into a project can be a very quick process for small files or a very slow process for large files if we use File - Link so that a local cache is automatically created, or if we use File - Create - New Data Source and choose cache.    But once a file has been linked Manifold's use of technology like MAPCACHE can make it very fast to view and to edit.   For all that if we bring data into a project by linking a non-Manifold file format into our project we will be limited by all of the limitations of that file's format.    See comments in the How to Edit a Single File topic.

Limitations on Linked Data

Linking a file into a project is different from importing because the data remains saved in the original format outside of Manifold.   When linking a file into a project the Manifold dataport doing the linking will do its best to provide as close a simulation as possible of the comparable Manifold component, for example, providing virtual tables for JPEG images that are linked so that it seems we are working with a Manifold image and not a JPEG.   But depending on the format being linked that only goes so far.   

 

For example, if we link an ESRI shapefile into Manifold it will look like a Manifold drawing but because shapefile format does not allow a mix of object times we will not be able to mix areas, lines and points within that drawing the way we could with a genuine Manifold drawing.  Likewise, if we link a CSV text file it will look like a table, but we cannot add an index to that table because CSV files are not database files and cannot store an index.  

 

Linking a data source into a project also creates a section of the project that is a portal to another world, where what we see in that data source is not in the project file but is a view into a different file or data source (like a database) which may or may not be writable.   See the discussion in the Portals to Other Worlds section of the Project Pane topic.

Cache and Linked Data

Cache in Manifold helps us get around some of the limitations of linked data.  

 

 

When linking an image or other data into Manifold the Link dialog provides a Save cache option at the bottom of the dialog that allows us to cache data from a linked file.   Using cache provides better performance and allows us to launch Assign Initial Coordinate System in the Info pane to specify the correct initial projection for linked data.

 

 

Using File - Create - New Data Source allows more sophisticated control over cache options.  

 

Caching data within the project will save a copy of data that is brought in from the external file within the project.   It is like a virtual import that, so long as the data within the external source has not changed, uses the data that has already once been brought into the project instead of re-fetching it again from the external source.   The cache will make subsequent use of the data very fast, but at the cost of a slow initial process, possibly minutes to link in larger data.

 

Checking the Save cached data between sessions box will save cached data within the .map project file so the next time that .map project file is opened what was cached from the prior session will still be available for use.   If the external source has changed since the last session, then cached data will be discarded as obsolete and fresh data will be fetched to refill the cache.

 

  Caution: When creating data sources based on web servers, like a Google image server, turning on the Save cached data between sessions option will save in cache a copy of every image tile fetched form the server.  That can result in astonishingly fast performance when next we browse that same region in the project, because all tiles will be local and will not need to be fetched over Internet from the server, but it will also mean that the size of the project can become very large (hundreds of megabytes or many gigabytes) very quickly.   Most Manifold users will leave that check box OFF unless they want to use that data when off-line.

 

Cache provides two primary ways of improving performance:

 

 

Using or not using cache is not a big deal for small data, like small images, but when images or other data are gigabytes in size cache can provide dramatic improvements in speed.  That is why the Cache data box is checked by default when creating data sources on regular file formats.  It is not checked by default when creating data sources on web servers.

 

Cache also provides some of the benefits of imported images.   For example, if the Cache data box is enabled we will be able to launch Assign Initial Coordinate System in the Info pane to specify the correct initial projection for linked data.   If we have turned off cache we will not be able to assign the initial projection.

Create a New Data Source

Creating a new data source establishes a dynamic connection to a data source, which could be a database server such as Oracle or PostgreSQL, a file database such as slqlite or a Microsoft .mdb file, a web server providing data using some protocol such as WMS or a Manifold imageserver module or even a plain, ordinary file format.    

 

Do not miss the use of image servers, as discussed in topics like the Example: Spectacular Images and Data from Web Servers example topic.

 

Creating a new data source using a file format is equivalent to using the File - Link command to link that file into a project (the capability is duplicated within the New Data Source dialog for user convenience) along with the option to connect in read-only mode.  When connecting to file formats new data sources are normally used with data sources based on .map projects, with file databases or with server-based data sources such as a DBMS or a web server source.

 

 Caution: By default data sources are opened in read / write mode if they are writable. When we edit data stored in a .map file the .map file is not changed until we do a Save, but in contrast, when we edit data in a data source such as a database system, any such changes made usually will immediately change the data in that data source without waiting for a Save.  That is expected behavior to database people but it may come as a surprise to someone used to a document editing model who may expect "nothing will change in the originating source until I do a save."

 

To protect data sources from inadvertent changes we can open them for read-only access by checking the Open as read-only box in the New Data Source dialog when creating the data source.  Opening a data source read-only in many cases will also allow other processes or users to simultaneously also open those data sources read-only to view them at the same time we view them, if the data source allows such multi-user, read-only access.

 

When we create a new data source Manifold adds the data source to the project pane using a gray cylinder icon to indicate it has not yet been opened. When we open the data source, say, by double-clicking on it,  Manifold begins scanning it using background threads and changes the icon to a yellow cylinder. Some data sources, like large Enterprise class database servers, can contain hundreds or thousands of tables and it can take some time to scan the data source to collect their names for display in the project pane.

 

 

During the scanning process a small triangle, like the "play" symbol on audio players, appears in the lower right corner of the data source yellow cylinder icon.  

 

 

When scanning is done the triangle scanning sub-icon goes away.  Scans are often so quick we might never notice the scanning sub-icon.  A yellow cylinder icon in the project pane for the new data source indicates the data from that source is stored outside the project within the data source.

 

 

If we create a data source that is read-only the icon will include a small black lock to indicate we cannot change data within that data source.

 

Note: In a Manifold project the System Data folder contains the two special mfd_meta and mfd_root tables that Radian uses to operate the project.  When we link data into a project or when we create a new data source from some non-Radian file, database system or other data source, Radian will show a System Data folder within that link or data source that contains mfd_meta and mfd_root tables as if they existed within the link or data source.  However, those tables do not actually exist within the non-Radian link or data source.   

 

In the case of such linked sources they are synthesized as virtual tables by Radian for Radian's use in operating that link or data source.  They are visible only to Radian and can be used only within Radian.  They do not actually exist in the data source and the data source's native query engine, if one exists, will not be able to see them.  For example, if we link a shapefile into a Radian project we won't create two tables called mfd_meta and mfd_root within any of the files that make up a "shapefile."

 

Radian will use the System Data folder for a linked data source to save other housekeeping infrastructure.   For example, Radian dataports for WFS, WMS, WMTS, OSM and other web servers which cache data will create cache tables within a Cache folder within the data source's System Data folder.

Advantages and Disadvantages of Imports

As convenient as it is working with data "in place" in whatever format or source it was originally, formats and sources vary widely in their technical characteristics such as speed, data capacity, robustness in the face of network or Windows glitches or ability to handle parallel processing that lets you use the full power of your computer.  

 

Importing data into Manifold avoids such problems, but it comes with some advantages and disadvantages.

Two Disadvantages

Importing data into Manifold has two disadvantages:

 

 

Many Advantages

In exchange for the above two disadvantages, importing data into Manifold has many advantages:

 

 

 

 

 

 

 

 

 

 

 

In an era when disk space is basically free but our time is more valuable than ever, it makes sense to trade off slightly larger files to get the above benefits.   It also make sense to tolerate slow imports of very large files on a one-time basis.  Most files import quickly enough.    Experienced Manifold users tend to move as much of their data as possible into Manifold project files, using .map files to archive data that originally may have been in other formats.

An Import Strategy

Tech Tip:  When you first install Manifold System, begin a systematic plan of converting your larger files into Manifold format during off hours.  Set a reminder on your computer to import a file just before lunch hour and just before your work day ends.  When you return, save the project in Manifold .map format.  Do that on a regular basis and before you know it, all of your big files will have been imported into native Manifold storage, ready to open instantly.

 

Apply the same plan to link into a Manifold project any files like ESRI shapefiles or MapInfo files you may some day want to edit "in place."  Start linking them into your project when you will be away from your desktop and then save the .map file when you return.  That will build a .MAPCACHE file, a potentially long process for a big shapefile.

 

If in advance you systematically import or link big files and then save the results as Manifold .map files, when you need to work with that data you'll be able to open the project instantly and work with lightning speed.   Enjoy life more.  Do your imports in advance.

Projections / Coordinate Systems

The terms projection and coordinate system are used as synonyms by Manifold.   

 

Getting it right on import:  When importing data, Manifold will use whatever coordinate system the source says should be used for that data.   Most modern formats used in GIS will automatically specify the coordinate system to be used.   But older formats won't do that, so we have to assign the coordinate system when we import from those formats.

 

The Info pane tells us at a glance if a coordinate system has been assigned OK.   When a data source specifies the coordinate system to use, Manifold displays the coordinate system in black color.  If the coordinate system is shown in red color, we must click the coordinate system picker button and choose Assign Initial Coordinate System.

 

 

 

Manifold shows the coordinate system in black color, as in the Latitude / Longitude coordinate system seen above at left, when the component has been created with coordinate system information or if it has been imported from a format that specifies the coordinate system to use.   When a component has been created without specifying the coordinate system or it has been imported from a format that does not specify projection information, Manifold will show the default Pseudo Mercator coordinate system as a placeholder in red color.   

 

Red color in a coordinate system name warns us that we must assign an initial coordinate system:  Click the coordinate system picker icon and use the Assign Initial Coordinate System dialog to specify what projection should be used.  Do that immediately after import whenever a coordinate system appears in red text.  

 

Tech tip:   What projection is used by data can be difficult to determine sometimes.    When downloading data from the web or other sources, always look for any accompanying notes or metadata information that says what projection is used for the data.  If we have no idea what projection is used and the file is older data, a good guess would be Latitude / Longitude.   For example, shapefiles that do not have a .prj file attached to specify what projection they use are often in Latitude / Longitude projection.

 

For a complete, step-by-step example, see the Example: Assign Initial Coordinate System topic.

Notes

Must be able to open the file -  When importing or linking a file our Manifold session must, of course, be able to open that file.  If something about Windows permissions or policies does not allow our login or the applications we run to access a file, then, of course, we will not be able to either import the file or to link to it.  Likewise, if some other process has locked out access to the file we will not be able to import it or link to it.  As a general rule of thumb, if some other process, including a different Manifold session, has opened a file read/write we will not be able to import from it or link to it.  If some other process has opened a file read-only, then usually we will be able to import from it or link to it, so long as we link to it read-only as well.

 

Large Imports take time - Importing data from big files can be slow while the data is converted into the special form Manifold uses internally to support fast performance.  Manifold cares about making work within Manifold as fast as possible, since that is what Manifold users spend most of their time doing, not about making faster round trips for import, edit and export.  Once imported data gets into Manifold it can be worked with astonishing speed, almost certainly much faster than in its original environment.  It can be saved as a Manifold .map project that thereafter can be opened instantly and saved instantly.  But to get that speed we pay a one-time cost of the time it takes to convert the data into fast Manifold .map form.   If we only intend to do a one-time edit it could be faster just to leave the data in place and edit it as a linked file, if possible.

 

Linking Large Files also takes time - When linking a large file, by default Manifold will build a MAPCACHE accessory file to enable faster viewing and editing of the linked file.  That can take a long time for large files but only the first time: once a .MAPCACHE file has been built thereafter linking that same file will be very fast.  Whether it is quicker to edit a single, large file by linking or by importing is often a toss-up, the choice being made on whether it is more important to edit the file "in place" because the file may be used by other programs that expect the original format.  To avoid building a cache, instead of using File - Link to link a file, use File - Create - New Data Source to link a file, since the New Data Source dialog provides options to not create a cache.

 

The origin of imported components -  We can get the path to a file that was imported by looking at the Log Window, which logs the path used for import operations.   

 

2021-06-29 12:01:32  -- Import: C:\data\OSM Venice\shapefiles\wiki landmarks.shp (0.007 sec)

 

That can be convenient, since text can be copied from the Log Window, and then pasted into any comments components giving the origin of a particular layer.

 

The origin of linked data sources - Hovering the mouse over a linked data source will show the path to that file in a tool tip.  Right-clicking the data source and choosing Properties will also show the path, but in JSON form that escapes backslash characters:

 

{ "Source": "C:\\data\\OSM Venice\\ESRI Venice.map" }

 

We can also get the path to a file that is linked by looking at the Log Window, which logs the path used for link operations.   

 

2021-06-29 14:39:13  -- Link: C:\data\OSM Venice\ESRI Venice.map

 

Videos

10 Minute Tutorial - Drag and Drop

 

See Also

Getting Started

 

Projects and .map Files

 

Exporting

 

User Interface Basics

 

Manifold Server

 

File - Import

 

File - Link

 

File - Create - New Data Source

 

Assign Initial Coordinate System

 

Where Manifold Data is Stored