GDB, ESRI File Geodatabase

Esri utilizes several different "geodatabase" formats for data, in four main types:

File geodatabases - Stored as files and folders within a file system. These exist in two versions, the old format used by ArcGIS 9 and the new format used by ArcGIS 10 and subsequent. Old format file geodatabases are rarely encountered. This topic discusses use of the new format file geodatabases. For discussion of old format file geodatabases, see the GDB, ESRI File Geodatabase, Old Format topic.

Mobile geodatabases - Esri mobile geodatabases are SQLite databases with specialized ST_GEOMETRY types. See the GDBmobile, ESRI Mobile Geodatabases topic.

Personal geodatabases - Stored in Microsoft Access file format and limited to 2 GB. Personal geodatabases are now a legacy Esri format. They have been replaced with file geodatabases in most uses. For information on personal geodatabases, see the MDB, ESRI Personal Geodatabase topic.

Enterprise geodatabases - Also referred to as "SDE" style geodatabases and stored within a database server such as Oracle or SQL Server. When connecting to popular databases like Oracle or SQL Server, if there is an Esri SDE installation in there, Manifold will automatically be able to use that as well. See the ESRI SDE Geodatabase topic.

File Geodatabases

Esri's file geodatabase format is the new ArcGIS "geodatabase" format that has replaced personal geodatabases and, in the new format, has replaced the now-obsolete "old" format file geodatabase used with ArcGIS 9. Data in a file geodatabase is stored and managed in a huge collection of files, all located within a single folder, with one of those files being named gdb (no extensions, just "gdb"). File geodatabases often called FGDB for short.

File geodatabases are recommended by Esri, and have become one of the most frequent formats used to publish spatial data for use by Esri applications. Esri provides an SDK/API, a software library with an application programming interface, to connect to Esri file geodatabases for read/write access. Manifold uses the Esri API to connect to Esri file geodatabases, to connect to vector (drawings) data and tables.

Esri's API does not allow access to rasters in file geodatabases. However, if we would like read-only access to rasters in file geodatabases we can use Manifold's ability to connect to file geodatabases through GDAL, an optional open source package that can be installed and used together with Manifold. Recent versions of GDAL include an experimental module that can connect to rasters in file geodatabases.

Manifold products like Release 9, Viewer, and SQL for ArcGIS Pro use Esri's own API to connect to file geodatabases, with the API built into the Manifold product. All required Esri DLLs for new format file geodatabases are included within Manifold installation packages, including portable installations, and are installed automatically when the Manifold product is installed. Launching the Manifold product thus always includes Esri file geodatabase capability, using Esri's own code. There is no need to install anything extra to allow Manifold connections to Esri file geodatabases.

The Manifold GDB Dataport is Read / Write

Some applications from other vendors that connect to file geodatabases are read only. The Manifold GDB dataport is fully read/write and allows Manifold to edit Esri file geodatabases in place. Manifold includes numerous technical features to facilitate read/write use of Esri file geodatabases without problems.

Geodatabases often contains some fields that are read only. Fields in a table window that cannot be edited will be shown with a gray background color. Examples are special fields like the OBJECTID field for the primary key, or calculated length or area fields that may appear in attribute tables. Fields with a gray background color are read-only fields and are typically calculated or otherwise automatically generated fields.

The Manifold GDB dataport:

See the Third Party Release Levels topic for the version of Esri's API currently used.
Allows creating, renaming, and deleting components of all types, storing component definitions inside the geodatabase.
Supports storing Manifold MFD_META properties inside the geodatabase. This allows overriding coordinate systems, setting up styles, and other Manifold features within the GDB.
Supports creating Manifold virtual computed fields within a GDB table.
Allows inserting, deleting and updating records in GDB tables.
Honors field length limits for GDB text fields. Attempting to write a text value into a GDB that is too long shows an error message with the name of the field, the maximum length allowed by the field and the actual length of the value.
Supports and utilizes Esri spatial indexes on vector data.
Supports datetime fields.
Exposes btree indexes on GDB data.
Refers to components using their query names whenever possible.
Exposes coordinate system data.
Reads multi-patch geometry.
Reads geometry with curves, including circular arcs and ellipsoidal arcs.
Reads and writes geometry with Z values.
Forces index names to be different from field names.
Re-uses API objects when reading data from GDB tables, providing higher performance.
Automatically adjusts variants of produced BTREE indexes for unique / nullable field properties.
Forces use of unique indexes on OID and similar special fields in GDB tables, avoiding problems caused when the API sometimes fails to describe them as such.
Logs errors returned by the API if the geodatabase fails to open.
Caches data for btree indexes on the OID field on demand, for significantly faster searches on those indexes.
Verifies the type of geometry values to match the type specified for the table when writing geometry data to a GDB table. If the type of a new geometry value does not match the type of values in the table, the write fails.
Optimizes memory used during reads.
Export to GDB adds zero Z to geoms that do not have it.
Formats a numeric fields and text fields with coded values domains using a choice style.
Handles renames and deletes of components and changes to table schemas.
Supports refreshing data from the Project pane (right click on the GDB data source and choose Refresh). This is useful for picking up changes to the geodatabase made outside of Manifold.
Supports autogenerated fields, such as OBJECTID.
Supports non-NULL constraints.
Translates coordinate systems with WKID values matching the correspondent EPSG code to EPSG:xxx to let Manifold enable coordinate transforms. Small differences between the definition of the coordinate system used by the database and the definition of the EPSG code used by Manifold are put into the coordinate system overrides.
Attaches EPSG codes to coordinate systems with equivalent WKID codes, to allow using grid-based and other EPSG coordinate transforms when re-projecting components with these systems.
The GDB dataport attempts to work around an Esri bug that frequently occurs in geodatabases created by recent versions of ArcGIS Pro. The bug makes a newly created table fail to register completely. The dataport examines the state of the geodatabase and, if the table has been created and is operable, reports that the table has been created and marks it in the metadata to prompt execution of special workaround logic within Manifold that compensates for the table not being registered completely in the geodatabase.
Deleting a table in a GDB from Manifold attempts to work around an issue in Esri geodatabase code that can cause a table to become unregistered in a geodatabase when a delete fails. The problem occurs when a table that is being deleted is locked in ArcGIS Pro. In that case the delete cannot succeed but it still starts and fails mid-way after unregistering the table from geodatabase structures. If the delete fails, the dataport checks the status of the geodatabase and if the table remains operable, the dataport marks the table in the metadata to trigger execution of special logic within Manifold that compensates for the table being partly unregistered.
The GDB dataport does not report Esri indexes on tables returned by file GDB modules as BTREExxx, as Esri's indexes can only be efficiently used to search for individual records, with other operations being unoptimized. This lets the query engine optimize around not having these indexes instead of being tricked into using them with subpar results. Built-in Esri indexes on the OBJECTID field are reported as BTREE, with all operations optimized internally by the dataport.
Exporting a drawing to GDB does not create a feature data set. Creating a feature data set for each drawing, for example, when exporting a map with multiple layers, is not necessary and often is not desired by the user. Manifold now only creates a feature class and leaves adding any desired feature data sets up to the user.
We can add new fields to an GDB using either the Schema dialog or using SQL. Creating a new text field in a GDB sets the maximum length of that field to 4096 characters.

Limitations of the Esri API:

The Esri API supports opening file geodatabases created with ArcGIS 10, but not "old format" ArcGIS 9. See the GDB, ESRI File Geodatabase, Old Format topic.
The Esri API does not support rasters (Raster Dataset, Raster Catalog, Mosaic Datasets and Raster Attributes).
Spatial queries using the File Geodatabase API are limited to the envelope-intersects operator.
The Esri API is not thread-safe, so Manifold cannot utilize it for parallel computation. For maximum speed and parallel performance, copy GDB data into the Manifold project and do intensive work within Manifold storage.
The Esri API snaps coordinates of geometry objects to an internal grid, so the final coordinates of the written geometry values within the file geodatabase usually will differ slightly from the coordinates supplied by Manifold data.
File geodatabases cannot handle mixed geometry types in the same table. However, the Esri API allows mixing geometry types in the same table even though file geodatabases cannot handle that. For example, a point object can be added into a table that contains areas, with the API not throwing any errors during the write; however, the spatial index then without warning stops returning half of the objects in the table. To avoid such internal inconsistency by the API, Manifold rejects attempts to write mixed geometry types via the API.
When creating a table in a file geodatabase, Esri requires the geometry type be specified in a property. That is easy to do in Manifold SQL, since the Manifold query engine allows specifying properties for the fields of a table when that table is created. We can also specify properties for a table created by SELECT INTO, for example, using the syntax SELECT ... INTO t (PROPERTY ..., ...) ...; This allows using SELECT INTO to create tables on the fly in an Esri GDB to which Manifold is connected. For example, when creating a table within an Esri GDB if we want to store areas in that table we must specify the FieldGeomType.geom property to have a value of area (types can be area, line, point, or pointmulti):

--SQL9

SELECT [area], [code], [name], [geom]

INTO [gdb]::[states] (

PROPERTY 'FieldGeomType.geom' 'area' -- the default would be 'point'

) FROM [states];

Note that to create a layer in the GDB that other packages can read, besides adding the FieldGeomType.geom property to specify the shape type, we would also have to add the FieldCoordSystem.geom property to specify the coordinate system to be used, for example, 'EPSG:4326'. In the property names we would substitute the desired geometry field name for geom if some other name is used, for example, FieldGeomType.SHAPE if the name of the geometry field is SHAPE. Example:

--SQL9

SELECT [area], [code], [name], [geom]

INTO [gdb]::[states] (

PROPERTY 'FieldGeomType.SHAPE' 'area',

PROPERTY 'FieldCoordSystem.SHAPE' 'EPSG:4326'

) FROM [states];

For an example of connecting to an Esri GDB, please see the Example: Connect to an ESRI GDB File Geodatabase topic.

Importing a File Geodatabase

In this topic, we illustrate import of a file geodatabase containing information on a gas utility company in Naperville, Illinois.

Important: When importing a file geodatabase file the tables and queries that appear in the Manifold project are Manifold components with no further connection to the file geodatabase file from which they were imported.

There are so many files within a GDB folder that it makes sense to first set the file type filter to GDB Files, so only the gdb file we want appears. We can then double-click it more easily.

To import from file geodatabase format:

Choose File-Import from the main menu.
In the Import dialog browse to the GDB folder containing the gdb file.
Double-click the gdb file desired.
Everything found in that gdb file geodatabase will be imported into the project.

This particular .gdb database contains numerous drawings and tables, organized within folders, related to a gas utility company in Naperville, Illinois. We can double-click a drawing or a table to open it.

For a more interesting display, we create a new map using the TaxParcel.Shape drawing and Bing streets as a base layer.

The drawing we added shows tax parcels in Naperville. We have used the Style pane to color them using different colors.

Linking a File Geodatabase

Important: When linking a file geodatabase file the tables and queries that appear in that data source in the Manifold project stay resident in the file geodatabase .gdb file. They are .gdb components even though they may appear in many respects, for the convenience of the user, to be Manifold components. Personal geodatabase files linked into a Manifold project are fully read/write.

The Save cache button allows setting cache options. Most often when linking to a format like file geodatabase, we will ensure the Save cached box is not checked. Working with a linked file geodatabase will be faster if we check the box, but if we are going to cache data within the project we may as well simply import the file geodatabase and use full Manifold speed. We uncheck the box and then we press Link.

To link a file geodatabase file:

Choose File-Link from the main menu.
In the Link dialog browse to the folder containing the gdb file.
Click the gdb file desired.
Check or uncheck the Save cache box as desired.
Press Link. A linked data source will appear in the project.
Press the + icon next to the data source to expand the data source to see the tables and queries it contains.

That creates a data source called NapervilleGas.gdb that contains all of the drawings and tables in the file geodatabase. We can click on the + icon by the data source cylinder to expand the data source hierarchy.

We can display a drawing in a map above a Bing satellite image server layer, as we did earlier in this topic. In the illustration above we show tax parcels in Naperville, using the Style pane to color the TaxParcel.Shape drawing.

We can style drawings within a GDB but only for the duration of the Manifold session. If we close the project and then re-open it, the style we have applied to drawings within the GDB will be lost.

Saving Style with Linked GDB

Esri GDB as a format cannot save Manifold style information, so if we would like to save style applied to drawings in a GDB in a persistent way, we can create a drawing in the Manifold project that links back to the data source, but which can be styled because it is resident in the project. This is similar to the technique used in the Example: Style Applied to an Image Server Image topic.

We link the NapervilleGas GDB into our project to create a NapervilleGas.gdb data source (illustration at left above). We expand the resulting data source, click on the TaxParcel.Shape drawing to highlight it, and we press the Copy button on the toolbar (illustration at right above).

We could also simply press Ctrl-C for Copy.

Next, we Right-click into the Manifold .map part of the project pane, that is, outside of the NapervilleGas.gdb data source, and we choose Paste from the context menu.

We could also just click into the .map part of the project to move the focus there and then press Ctrl-V for Paste.

That creates a new drawing in the project called TaxParcel.Shape. Like all drawings, the new TaxParcel.Shape drawing does not actually contain any data. Drawings are simply a component that says what geometry field in what table to visualize.

We can right-click onto the new TaxParcel.Shape drawing and choose Properties to open the properties pane to see that the new drawing consists of just a few items of necessary information: the drawing uses the geometry field called Shape from within the table called [NapervilleGas.gdb]::[TaxParcel], that is, the TaxParcel table that is inside the NapervilleGas.gdb data source. The properties also include Style properties for the area symbology and the fill color (ColorBack).

Drawings are just a link, a shortcut to where the data resides, in this case a table within an external data source, along with some other properties, if desired, like style.

We can double-click open the new TaxParcel.Shape drawing to see it uses the same Style as the drawing within the NapervilleGas.gdb data source. That's no surprise since it was copied from that same drawing. Copying a drawing copies all of its properties, including Style properties.

We can right-click onto the TaxParcel.Shape drawing within the NapervilleGas.gdb data source and choose Properties. we can see that it uses exactly the same Style properties as the new TaxParcel.Shape drawing within the main part of the project.

The new TaxParcel.Shape drawing is in the Manifold .map portion of the project, so when we save the project the drawing and all the style properties it has will get saved, ready to be used the next time we open the project.

Copy and Paste between Manifold Sessions

Suppose we would like to utilize the styled tax parcel drawing we have just created in a different Manifold project. We can copy and paste components and data sources between different Manifold projects, so that is easy to do.

We have saved our project as NapervilleGas_gdb.map in Manifold .map project format. Perhaps this is just one of many Manifold projects we have saved on an archival server our workgroup or organization uses.

In the illustration above, we have opened our NapervilleGas_gdb.map in one Manifold session.

We would now like to use the nicely-styled TaxParcel Drawing in a new project. We launch a second session of Manifold, which opens with a new, blank project, seen above.

In the NapervilleGas_gdb project we Ctrl-click on the NapervilleGas.gdb data source and we Ctrl-click on the TaxParcel Drawing, to highlight both of them. Next, we press the Copy button on the Project pane toolbar (or, we simply press Ctrl-C for Copy).

We switch to the new, blank project session of Manifold and we press the Paste button on the Project pane toolbar.

The NapervilleGas.gdb data source and the TaxParcel Drawing appear in the project.

When we double-click open the TaxParcel Drawing we see it is the same drawing, retaining the same style.

Copying and pasting data sources between Manifold sessions works when the PATH to the data source resource is the same from both sessions.

Exporting / Writing to GDB

GDB drawings cannot handle a mix of object types. Therefore when exporting to GDB, Manifold will split a drawing that contains a mix of points, lines and areas into three different drawings within the GDB. Likewise, when writing to a GDB that is linked, for example, when copying a drawing from a Manifold project and pasting into a linked GDB data source, Manifold automatically splits a drawing containing geoms of different object types into separate drawings for each type within the GDB

Notes

Text value truncation - Exporting data to an Esri file geodatabase automatically truncates text values to 4096 characters, to protect against accidental insertions of very long text values that might impact geodatabase performance. When text fields are truncated, the log window will report the names of fields containing truncated values as well as the total number of truncated values in the log.

Pass through of GDB to third party applications - When an Esri GDB file geodatabase is connected as a data source within a Manifold project, that data source then becomes available to any third party application which connects to the Manifold .map project by using the Manifold ODBC driver. This provides the ability to read/write Esri file geodatabases even to those applications which on their own cannot connect to Esri file geodatabases. See examples cited below.

Mixed Geometry Types - Manifold allows a mix of geometry types, such as points, lines, and areas, within the same table, while GDB does not. Copying and pasting a table that uses a mix of geometry types from a .map file into GDB will automatically sets the geometry type of the pasted data to whichever type is used by most records. This is a way of ensuring that tables which use only one type (points, lines, areas) have the type that they use correctly specified in GDB.

Esri is not Manifold - When connecting to GDB we must be aware we are no longer using Manifold code but instead are using Esri code. Esri products are high quality and there are many advantages to using Esri's own code to connect to intricate Esri formats such as GDB, since presumably Esri's code can handle all the many internal and undocumented quirks of the format. One major disadvantage is that Esri's code in general is not as bulletproof as the Radian technology used in Manifold, so connections to GDB fall outside of Manifold's reputation for never crashing.

Rasters (Images) in File Geodatabases - To connect to Esri file geodatabases, Manifold's built-in connection to file geodatabases uses Esri code within Esri's API for connections to file geodatabases. Esri's API does not allow access to rasters in file geodatabases, so Manifold's file geodatabase data source will not show any raster (image) contents of a file geodatabase. If we would like read-only access to rasters in file geodatabases we can use Manifold's ability to connect to file geodatabases through GDAL, an optional open source package that can be installed and used together with Manifold. Recent versions of GDAL include a module that can connect to rasters in file geodatabases. The GDAL raster geodatabase module is not Manifold code and it is not Esri code. It was written by third parties who have attempted to reverse engineer Esri's internal raster geodatabase format. The module is not supported by either Manifold or Esri.

Videos

Manifold Viewer - View GDB Faster than ArcGIS Explorer - Esri's ArcGIS Explorer product is a free viewer that allows Esri people to view Esri GDB Geodatabases, along with other data. Unfortunately, Esri has stopped producing Explorer and has replaced it with ArcGIS Earth, which cannot connect to and display Esri GDB Geodatabases. The video compares Explorer to Viewer side-by-side so Esri users can see that if they need to continue viewing GDB Geodatabases for free they can reliably use Viewer for fast, high quality GDB viewing.