This topic provides release notes for the very latest builds. Release notes have been adapted from announcements published by Engineering representatives in the georeference.org user forum. SQL for ArcGIS Pro uses a common code base with Manifold's Release 9 series of GIS products: builds listed below are those which are of special interest to SQL for ArcGIS Pro users.
Build 177.2 is a historic build in that it drops support for 32-bit versions of Manifold projects. All Manifold products are now 64-bit and require 64-bit Windows operating systems. Build 177.2 supports ArcGIS Pro 3.x for SQL for ArcGIS Pro, and also adds features to various dataports and provides bug fixes.
Build 177.2 expands the display viewport size in table windows to 2 billion records. That is way more than anybody can read through, but together with Manifold speed plus reasonable safety measures now built in, increasing the number of records that can be interactively displayed in a table window greatly expands the reach of interactive procedures to very large tables.
SQL for ArcGIS Pro
SQL for ArcGIS Pro tries to locate manifold.exe using a dialog if it cannot be found automatically. The location of manifold.exe is then saved in an environment variable for future reference.
SQL for ArcGIS Pro supports ArcGIS 3.x. The install packages include two versions of the add-in: a version for ArcGIS 2.x in the ArcGis2 folder and a version for 3.x in the ArcGis3 folder. The EXE and MSI install packages automatically detect the installed version of ArcGIS and register the corresponding version of the add-in.
SQL for ArcGIS Pro no longer includes 32-bit modules.
Manifold products no longer use the bin or bin64 folders. DLL and EXE modules are placed directly within the install folder. The Shared folder has been renamed to Extras.
SQL for ArcGIS Pro no longer includes an Open SQL 32-bit button. The Open SQL 64-bit button has been renamed to Open SQL. Since ArcGIS Pro requires 64-bit Windows anyway, the deprecation of 32-bit versions has no significant impact on SQL for ArcGIS Pro users.
Discussion
Manifold was one of the first software companies to ship 64-bit products. Despite being one of the first to move to 64-bit, for over 20 years Manifold has maintained both 64-bit and 32-bit versions of all Manifold products.
Over time most, if not all, users naturally moved to primarily use the 64-bit versions of products. At the same time, maintaining obsolete 32-bit code has become a limiting factor for adding new features. Manifold is therefore going exclusively 64-bit from build 177.2 onward. That will make it more efficient to add new features, and will remove limits on how those new features must be designed. That will make many things simpler for new users. The first example of such a new feature is providing 2 billion record displays in table windows, which would have been limited by 32-bit restrictions. Going exclusively 64-bit also reduces the size of the SQL for ArcGIS Pro download to less than 31 megabytes.
There are two areas in which SQL for ArcGIS Pro users may be relying on 32-bit code directly or indirectly:
The maximum number of records that can be shown in a table window is now 2 billion. Tables from external data sources can contain more records, but only up to 2 billion can be displayed for interactive browsing in a table window.
The table window can either show all records in a table (up to 2 billion) or show a first sample of records. Showing a first sample of records instead of all records is useful when working with external data sources like a file geodatabase, or, say, SQL Server: not fetching all records immediately protects against wasting server resources and network traffic needlessly. Tables from .map files show all records by default, while tables from other data sources show a first sample of records. This also applies to the result tables of queries, including ad-hoc queries in command windows. The "show a sample from non-.map data sources" is not absolutely bulletproof. For example, a query stored in a .map file may access records in a table stored on SQL Server. When the user runs such a query, the table window will try to fetch all records in the result table, because the query itself is stored in a .map file, even though the records come from SQL Server. In cases like this, the responsibility for not overloading the server for no good reason is on whoever writes or runs the query.
The Tools - Options dialog includes an Initial number of records to show (non-.map) option to specify the size of a first sample of records fetched for non-.map data sources. The available values are: auto (default), all, or a specific number of records. The auto value is currently set to 5000 records. The previous default was as high as 50000 records because it applied to tables in .map files as well. 5000 records is much friendlier to database servers. If 5000 records is too low for the data sources you are working with, you can always change the option to a higher value in the Options dialog.
The Info pane shows the number of fetched records in the table window. If there are more records available, the Info pane also shows a (+) notation. If the table window is showing a first sample of records instead of all records, the Info pane also shows a button to fetch all records. Once a table window has been told to fetch all records for the displayed table, it will keep fetching all records for all further operations that refresh or replace the table.
The Info pane shows the number of displayed records in the table window. That number will be the same as the number of fetched records that pass any filter.
The Info pane shows the Component tab for a command window with a table.
Table windows detect changes to displayed data from a non-.map data source and show an action bar below the record list with a prompt to refresh data manually. To refresh data, use the View - Refresh Data command or click the prompt in the action bar.
A table window displaying data from a .map file does not automatically refresh data after changes to the data if the number of records in the table exceeds 64k. Instead, the window shows an action bar with a prompt to refresh data manually. For example, if a drawing's table is open and the table shows 100000 records, and in the drawing some of the object are deleted, the table will not automatically refresh. Instead it will show an action bar with a prompt to refresh. That allows convenient interactive editing of drawings while their table windows are open without the table window (which might display up to 2 billion records) refreshing itself on every small change in the drawing.
A table window showing over 64k records detects deleted records and shows them in gray background color, with an X icon on the record handle. For example, deleting objects in a drawing window that has 100000 objects when the drawing's table is open will show empty gray rows in the table for the deleted objects. Pressing the refresh prompt in the action bar will refresh the table so those empty rows will be deleted. This provides fast means to delete objects in maps without fully refreshing tables, while still showing which records have been deleted.
The context menu for the fill record in the table window no longer includes a Stop Reading Records command.
The table window always applies filters only to the fetched records. The Filter Fetched Records Only command has been removed. Now that table windows can show up to 2 billion records, always using only the fetched records is no longer a limitation in the vast majority of cases. Tables bigger than 2 billion records can still be filtered using a query composed with the Filter using Query command.
Table windows always apply sort order only to the fetched records. If the table window shows a first sample of records instead of all records and you want to sort the entire table, fetch all records first. The maximum allowed size of field values used for sort ordering is 16 GB.
Watch the new 10 Minute Tutorial - Very Big Table Windows video to see 1.295 million records displayed in a table window. That video shows Maniofold Release 9, but table windows work the same way in SQL for ArcGIS Pro. Fast!
Failed attempts to connect to a data source are recorded. Subsequent attempts to connect to the same data source are throttled. The delay starts at 1 second and gradually increases to 15 seconds. Errors for failed attempts to connect to a data source are logged with repeat errors (same error message) omitted.
Exporting ECW or JPEG2K tracks progress and can be canceled.
Exporting ECW or JPEG2K aligns reads to tile boundaries for better performance.
Exporting ECW or JPEG2K automatically adjusts internal buffer sizes when exporting very big images for performance (and, in very rare cases, to allow the export to succeed where it previously was failing due to exceeding one of the internal limits on temporary data).
Reading XYZ with a non-space delimiter allows whitespace both before and after the delimiter.
Reading XYZ with integer values that do not fit into INT32 converts all values to FLOAT64.
Reading XYZ skips NaN values. Records with such values have no business being in the file, but they might technically be there and some software packages apparently produce files with them.
Reading XYZ tracks progress and can be canceled.
Reading XYZ supports files bigger than 4 GB.
Reading XYZ protects from tiny drift in XY values to handle regular grids more reliably.
Reading XYZ / NC performs significantly faster, 4-5 times faster, and even faster with big files.
Writing XYZ performs significantly faster, 5-6 times faster, and even faster with big files.
(Fix) Exporting the result of a query with computed fields to MDB, SQLITE, or similar formats no longer fails.
(Fix) Printing geometry to WKT prints points with Z values as POINT Z or MULTIPOINT Z instead of POINT orMULTIPOINT.
(Fix) Printing geometry to WKT no longer ignores Z values for areas.
(Fix) Reading FLT with a malformed HDR may no longer erroneously set pixel scales to zero.
(Fix) Attempting to use components from a data source that fails to connect, for example, because it is referencing a file that does not exist, no longer disrupt the user interface, such as map windows, layout windows, or panes.
(Fix) Reading XYZ no longer ignores the first record if the file starts with UTF8 BOM.
(Fix) Reading NC creates an RTREE index on tiles.
(Fix) Writing XYZ prints all values with full precision. Previously, values that were very small or very large could lose some of the digits.
Build 177.1 extends interactive selection capabilities, makes it easy to switch licenses, and provides bug fixes as well as improvements in the MapInfo MIF dataport.
The Info pane shows the number of selected records in a table window (this includes table windows opened for query components, but not command windows for ad-hoc queries) or for the current layer in a map window, if that number is known.
There are two scenarios in which the number of selected records is unknown:
The total number of records in a table is unknown and the selection is inverted. This happens with streaming tables. Prime examples are the result tables of queries: such tables compute their data as they are being read so that you can start working with the returned records without waiting until all records are computed. An inverted selection stores keys of the unselected records, so while the system knows how many records are unselected, it cannot compute how many are selected.
The total number of records in a table is known, but the table has changed in a way that could affect the selection after the last selection operation. Examples of changes that can affect the selection are adding or removing records, or changing values in the key fields.
If the total number of records is unknown, the number of records is shown as a question mark, ?. If the number of selected records is unknown, the number of selected records s shown as a question mark, ?. If either of these numbers is unknown, the Info pane shows a Refresh button next to them. Pressing the Refresh button will scan the table and update numbers that are currently unknown. The scan tracks progress and can be canceled.
Combining the results of a selection template with an existing selection performs faster.
The Help - About dialog includes a new Switch License button that allows switching the current license to a different serial number. The dialog prompts the user to restart the application after the license is switched.
Writing MapInfo MIF format now exports xN and UUID fields as CHAR data type.
(Fix) The Clear Filter command in a table window no longer fails to uncheck the selection filter button in the toolbar.
(Fix) Copying a table with mixed geometry into a database (eg, PostgreSQL) using copy and paste or drag and drop no longer sometimes fails because the new geometry field is wrongly limited to a specific geometry subtype.
(Fix) The Spatial select template that selects data based on a spatial overlay no longer fails to clear the selection when the overlay is empty and the selection mode is replace or intersect.
(Fix) The result tables of SelectionXxx query functions no longer sometimes fail to return data.
(Fix) Reading MapInfo MIF format no longer sometimes ignores the last object.
(Fix) Reading MapInfo MIF format correctly parses DATE, DATETIME, and TIME values.
(Fix) Writing MapInfo MIF format exports INT16U as INTEGER data type (was SMALLINT) and INT32U as FLOAT data type (was INTEGER), to avoid potential data loss.
Build 177 is a new base build, incorporating all improvements in the 9.0.176.x series of builds. It is recommended for all users.
There is a new style for text fields in tables: file = absolute path to a file. When text fields are formatted using that style, table windows and the Info pane will highlight text in blue color and will show a new Open File command in the context menu when a cell is clicked. If a specific value is a well-formed path, but the file referenced by it does not exist, the Open File command is disabled. The Open File command opens the file using whatever is the default Windows handler for that file type. That makes it easy to have fields in tables that contain file names that are images, PDF files, saved RDP files, or anything else that Windows can open, and to be able to open those files instantly from within Manifold.
Build 176.7 delivers small improvements to the user interface that are of interest to SQL for ArcGIS Pro users working in Advanced mode:
There is a new View - Panes - Fit to Screen command that puts all undocked panes and windows onto the screen. If there are multiple monitors, the command chooses the nearest monitor for each pane or window and puts the pane or window onto that monitor. The command helps locating panes and windows that have been accidentally moved nearly offscreen using the cursor or ended up being completely offscreen due to changes in monitors (may happen with remote desktop connections, among other scenarios).
The Register pane shows automatically placed control points with a distinctive icon. Any editing done to an autoplaced control point removes the autoplaced flag.
Alt-clicking a control point in the Register pane pans the map window to that control point.
Creating a new RTREE index for tiles suggests using fields starting with x and y for XY values.
Editing a traverse allows entering a traverse command for the insert coordinate. This allows entering geometry data as a sequence of traverse commands, with visual feedback at every step.
(Fix) The TileGeomOverlayCount query function on longer allows specifying a non-numeric field as an argument. Either specify either no field by passing an empty string, or specify a numeric field.
Build 176.6 introduces dramatically faster spatial overlay functions and faster LiDAR performance with LAZ files, as used in Advanced mode. Big jobs that previously took over two and a half hours now can be done in six minutes.
GeomOverlayXxx (non-topology) query functions can be canceled in the initial data collection phase.
GeomOverlayXxx (non-topology) query functions optimize cases where a big source table has a small overlap area with an overlay table, as long as the source table has a spatial index.
GeomOverlayXxx (non-topology) query functions optimize cases where a source table has a small overlap area with a big overlay table, as long as the overlay table has a spatial index and the source table can determine its bounds without being read in full (this usually means that the source table also has a spatial index and is static, that is, not a query).
GeomOverlayXxx (non-topology, non-filter) query functions now only return pairs of records for the actual overlay incidents. Previously they were also returning a single record for each record in the source table which had no corresponding records from the overlay table, similarly to a LEFT JOIN. We changed the behavior to be similar to an INNER JOIN, because otherwise the functions have to return at least as many records as there are in the source table, which is frequently undesired and just makes processing the result slower. If there is a need to obtain records from the source table that have no overlay incidents, that can be done separately after.
GeomOverlayXxxFilter query functions perform proportionally to the number of records that have overlay incidents. Previously, fetching all records of the overlay filter required going through all records of the source table. This and other changes to GeomOverlayXxx functions apply to various UI tools that use them, eg, they apply to the spatial overlay select templates and to the Join dialog when it transfers data between drawings.
SelectionXxx query functions perform proportionally to the number of selected records. Previously, fetching all selected records of a table required going through all records of that table. The change applies to various UI tools that work with selections, eg, copying a few selected records from a large table now performs much faster. The only exception is that selections that have been inverted continue to perform proportionally to the size of the table, not to the size of the selection.
Opening a LAZ file with a COPC index validates index data and turns off the index if its data is invalid.
Reading a LAZ file no longer progressively caches uncompressed data on disk and instead uses multiple reading heads. This significantly improves performance.
Performing a spatial search on a LAS or LAZ file with a COPC or .mapindxp index works faster.
The above improvements significantly improve the performance of spatial overlays when input data sets are not similar in size or do not significantly overlap. That is a very common situation since often a smaller data set, such area objects for a particular town or parcel, is used to extract data of interest from a very much larger data set, which might cover an entire county or province. The exact performance gains depend on the overlap size, and can easily be 5x, 10x or more. For LAZ, the improvements are even bigger. A test example used a LAZ file 375 million points with a COPC index and fetched 3.5 million points covered by an area in a different drawing using GeomOverlayTouchingFilter. The time to complete the job in prior builds was two hours and 38 minutes (9230 seconds). Using build 176.6 the time dropped to only six minutes (363 seconds), 25 times faster. If the area was even smaller, extracting fewer points from the LAZ file, the task would have taken even less time.
(Fix) Selecting records in a table with an identity index on text fields with external collates (for example, as in an external database) no longer sometimes fails to work.
(Fix) Writing a BIL, FLT, GRD, IMGERDAS, or TIFF file with pixel scale expressed in units other than meters no longer writes wrong local offsets to the accompanying .MAPMETA accessory file.
(Fix) Rendering a LAZ file with a COPC index no longer sometimes prunes search too aggressively.
Build 176.5 expands the license activation system to add an option for enabling installation of a license for all users on a machine at once. That simplifies use for some organizations, for server use, and for IMS use in Release 8, which uses the same activation system.
Build 176.5 also introduces a new, integrated, parallel Download Manager, a new, high speed, parallel system for downloading tiles from web servers such as image servers, TMS servers, WMTS servers and ArcGIS REST servers when working in Advanced mode. The download manager is a built-in feature within SQL for ArcGIS Pro that works automatically in background. There is no need to install, configure, or administer any modules or other software. Just use SQL for ArcGIS Pro the way you always do and web servers seem to be much faster and more responsive.
The build also provides Advanced Mode conveniences such as persistent data types in the Schema dialog remembering the last-used type, Drag and Ctrl-Shift-Drop to link data sources read-only (which facilitates creating projects for shared use), a new email style in tables for rapid emailing, and a Suggest Locations toolbar button in the Register pane for faster georeferencing. A variety of other features and a few fixes are also in the new build.
SQL for ArcGIS Pro licenses optionally can be installed (activated) for all users on the same system. The About dialog shows whether the license in use has been installed for all users, and it allows installing or removing the license for all users. Installing or removing the license for all users requires administrative privileges and will prompt the user for administrative credentials.
The Activate dialog includes an option to install (activate) an SQL for ArcGIS Pro license for all users on the same system. The option requires administrative privileges and will prompt the user for administrative credentials.
The download manager caches downloaded data in an application-wide cache which is only cleared when the running application, such as SQL for ArcGIS Pro or a third-party application using Advanced Mode facilities through the scripting API, is closed. That allows the system to reuse downloaded data between multiple dataports connecting to the same server even across different MAP files.
The download manager allows the user interface to cancel outstanding downloads in the middle of those downloads without waiting for the network. This prevents freezes when a map window that shows a web image is getting panned or scrolled or closed in cases when requests for tiles being downloaded could take a long time to determine if they will complete or fail.
The image server dataport uses the new application-wide download manager.
The TMS dataport downloads tiles using the new download manager.
The WMTS dataport downloads tiles using the new download manager.
The ArcGIS REST dataport downloads tiles using the new download manager. Note that ArcGIS REST servers can be configured to serve some images as screens rather than tiles. Such servers and images will work, but without the benefits of parallel downloads and reliable cancels provided by the download manager.
The Field dialog in the Schema dialog saves and restores the last used data type when creating a new field. This allows quickly adding multiple fields of the same type, for example, quickly adding a series of fields that are all text fields when altering a table's schema.
Drag and Ctrl-Shift-Drop - Dragging and dropping files into the Project pane with both the Ctrl and Shift buttons pressed will link those files in read-only mode. That makes it easy to create projects with read-only data sources, a great convenience when creating projects that will be shared by other users at the same time.
The Project pane shows disconnected data sources that are linked in read-only mode using a lock icon. That makes it easy to see which data sources are read-only even before they are used.
Text values in tables support a new style for email addresses. Email addresses are shown using the same color as URLs. Right-clicking an email address shows a new Send Email command in the context menu, which automatically launches your mail client, such as Outlook, with the email address loaded in a new email message. That makes it very easy to compose and send an email to the right-clicked address.
New query function: StringEmailValid - takes a string and returns true if the string is a valid email address. The validation is permissive and should only be used to filter out strings that are certainly not email addresses.
New query function: StringUrlValid - takes a string and returns true if the string is a valid URL.
The Manifold ODBC driver allows connecting to a new temporary database using DRIVER=...;DBTYPE=TEMP; in the connection string.
The SQL Server dataport supports DATETIMEOFFSET and DATETIME2 data types. DATETIMEOFFSET values are automatically converted to the timezone for the current user on both read and write. Reading a date with a timezone converts data to the timezone for the current user. Entering a date without a timezone will use the timezone for the current user.
(Fix) Importing a TIFF file no longer sometimes fails to create an RTREE index on x-y-tile fields (a regression).
(Fix) Alt-clicking a tile in an image in a linked TIFF file no longer sometimes fails to show pixel values.