The Tools - Index LiDAR Files command creates point cloud spatial index .mapindxp files for LAS and LAZ files that contain LiDAR point clouds. The Index LiDAR Files command scans folders and sub-folders of LAS / LAZ files and building indexes for them. The Index LiDAR Files command is available even if there is no project open.
If a LAZ file already contains a COPC spatial index, Manifold will recognize that spatial index and will automatically use it. When working with LAZ files that are in COPC form there is no need to run the Tools - Index LiDAR Files command to build an index in advance. There is no harm done in attempting to run the Tools - Index LiDAR Files command to create indices on LAZ files in COPC form: the tool will report the files already have indices.
Manifold Viewer
is a free version of Release 9 that is generally read-only. One
exception to "read-only" is that the Index
LiDAR Files command works in Viewer too. We can use the free
Viewer to create spatial indices for LiDAR files. Super!
The Index LiDAR Files command is intended to allow background indexing of LiDAR files while we continue work on other things. To use the command, launch a Manifold session and launch the command. Choose the folder (and thus subfolders) containing LiDAR files and allow that Manifold session to index them all. We can continue working in other Manifold sessions while the session running the Index LiDAR Files command does its work. Indexing LiDAR files takes time, so if they are indexed in background ahead of when they are needed, they can be quickly linked into Manifold projects when they are needed, either using the LAS/LAZ dataport or the librarylas dataport.
Launch Manifold and choose Tools - Index LiDAR Files.
Click the [...] browse button to launch the Select Folder dialog.
In the Select Folder dialog, Click the desired folder to highlight it, and then press Select.
The Index LiDAR Files will search that folder and all sub-folders to build a list of all LAS or LAZ files found.
Press Ctrl-A to select all files, or Ctrl-click individual files to select them. The usual selection moves within tables and grids also work.
Press Index to index all selected files. The process will automatically skip indexing file that already have current indices.
Data Folder |
Path of the desired folder. |
|
Press the browse button to navigate to the desired folder and to choose it. |
Allow parallel execution |
Process multiple files in parallel, using multiple threads. The maximum number of files indexed in parallel is 8 files for 64-bit Manifold. That provides parallelism for indexing speed while leaving reasonable resources for continued work in other Manifold sessions or other applications |
(display) |
For each LAS or LAZ file found in folders and sub-folders, the dialog displays the name and relative path to the file, the size of the file and the status of the file. Status is one of:
|
Index |
Create indices and metadata files for all files in the folders and subfolders. |
Close |
Close the dialog without doing anything. |
Spatial indices allow far faster work with spatial data, but spatial indices designed for general GIS work are not efficient with the very large numbers of individual point objects in a typical LiDAR data set. Manifold therefore builds a specialized index for point cloud data when linking to LAS or LAZ files individually or when using the librarylas dataport to create a data source linked from a folder containing LAS or LAZ files.
The best way to build spatial indices for LAS or LAZ files is to run the Tools - Index LiDAR Files command to create indices in advance for LAS or LAZ files we intend to use. The Index LiDAR Files command runs in parallel for faster performance and can index multiple LAS or LAZ files simultaneously. It is good practise immediately upon downloading LAZ or LAZ files to launch the Index LiDAR Files command to index those files. That will create indices so when we link those LAS or LAZ files we will not have to wait around in real time while the system creates indices.
When using COPC LiDAR data, there is no need to create indices in advance since Manifold will automatically use the COPC index. Attempting to run the Tools - Index LiDAR Files command to create indices on COPC LiDAR LAZ files will report the files already have indices.
If we have not created indices in advance and we are not using COPC data,, when we link a LAS or LAZ file into our project, Manifold will automatically create an index for that file.
When no point cloud spatial index exists, the system builds the index is built the first time a LAS or LAZ file is linked into a Manifold project. That can take significant time, possibly minutes per file in the case of LAZ files (faster with LAS files), but the spatial index needs only to be created once. The point cloud spatial index that is created is stored in an accessory .mapindexp file in the same folder as the linked LAS or LAZ file. Manifold will automatically use that .mapindexp spatial index the next time the accompanying LAS or LAZ file is used in a project.
Whenever we copy to another location LAS or LAZ files that have been used with Manifold, we also should always copy along with them the accompanying .mapindexp (point cloud spatial index) and .mapmeta (coordinate system) accessory files Manifold has created, so those accessory files can be re-used for much faster launching of projects that use their LAS or LAZ files.
The specialized point cloud index is a recent addition to Manifold which has not yet been implemented for LAS or LAZ files that are imported into a Manifold project. That creates, for now, the unusual situation that Manifold can display and work with LiDAR data faster when linking a LAS or LAZ file into a project, than importing the file into the Manifold project. Usually, workflow is much faster when we import a file into Manifold; however, LiDAR point cloud data in LAS files, for now, is an exception to that general rule.
Upcoming Manifold builds will include the specialized point index within .map files as well, which will mean imported LiDAR data will be even faster than linked LiDAR files. However, for now we should link LiDAR files if we want maximum speed. A bonus of linking LiDAR data into a project is that the Manifold project stays small, typically only one megabyte in size, since all of the LiDAR data remains stored in the LAS file.
Manifold starts
building a point cloud index when we first
open either a linked LAS or LAZ drawing or the drawing's table.
The moment we click open the table, or we click open the drawing
or add the drawing as a layer to an opened map, Manifold will launch a
background process to build the index. Building the index in background
allows us to keep working, doing other things, while we wait for the index
to be built. Although the drawing window will open, no points will be visible
until the index is finished building.
We can avoid waiting around for the index to be built by running the Tools - Index LiDAR Files command to build an index in advance.
Manifold will read any coordinate system information specified by LAS or LAZ files. In addition, Manifold also will get projection (coordinate system) information from any sidecar PRJ or PAM XML files that accompany the LAS or LAZ files. If an accompanying file specifies a different coordinate system than the LAS file specifies, the sidecar file wins, since sidecar PRJ or PAM XML files are often provided to override, that is to fix, erroneous coordinate information specified within the LAS.
We will index a folder full of LAS LiDAR files.
Launch Manifold and choose Tools - Index LiDAR Files. That launches the Index LiDAR Files dialog.
Press the browse button to launch
the Select Folder
dialog.
In the Select Folder dialog, navigate to the location of the folder that contains the LiDAR files of interest. We would like to use LiDAR files in the Golden Gate folder and any sub-folders within. We click on the Golden Gate folder to load it into the Folder box, and then we press the Select button.
That loads the folder we chose into the Data Folder box. Manifold traverses that folder and recurses into any sub-folders within, to find any LAS or LAZ files. In this case, the Golden Gate folder we chose has no sub-folders, and it only has four LAS files, There are no LAZ files in the folder, but if there were the dialog could handle them as well. The dialog reports that none of them have a Manifold point cloud spatial index, that is, none of the files have an accessory .mapindexp file.
To select all of the files, we press Ctrl-A. We could also use any of the usual selection methods for tables or grids, such as Ctrl-clicking files to select them, or Ctrl-clicking the first of many files and then Shift-Ctrl-clicking the last of many files to select both files and all in between.
Press Index to index all selected files.
The system swings into action and commences indexing all selected files, indexing eight files in parallel in 64-bit Manifold. A dialog provides status. We can cancel the operation at any time.
With few files to index the process will finish quickly. In cases where there are dozens or hundreds of LAS or LAZ files, we would leave the Manifold session running the dialog to do its work while we continued with other projects, perhaps launching other Manifold sessions to work with Manifold while the session running Index LiDAR Files continues to cook. If we have many files to index, we can leave the session running overnight.
When the system finishes, it reports the number of points within each file. That number is taken from the .mapmeta accessory file the dialog automatically builds in addition to the .mapindexp spatial index file. We press Close.
Now that we have built point cloud spatial indices for the LAS files in the Golden Gate folder, those files will be especially quick to use whenever we want to create a data source with them, either using the LAS/LAZ data port or the librarylas data port.
Suppose at some future point we want to link those files into our project using the librarylas data port.
We choose File - Create - New Data Source.
We enter a Name of Golden Gate to remind ourselves what the data source contains. In the Type box we choose Folder: librarylas. We browse to the Golden Gate folder and choose that folder. Press Create Data Source.
The data source opens instantly, and when we drag and drop the drawing inside the data source into a map, the points appear instantly, with no delay from creating point cloud spatial indices for the LiDAR files in the Golden Gate folder. Those have already been created and are ready for use.
Regrettably, it is routine to encounter LAS/LAZ files that wrongly specify coordinate system axis ordering of YX when in fact the data uses XY axis order. Such errors are a classic case of wrongly specified coordinate systems discussed in the That YX Thing essay.
To date, no LAS/LAZ files have been encountered that specify YX axis ordering and which in truth use YX axis ordering. Therefore, the Manifold LAS, LAZ LiDAR and librarylas dataports as well as the Index LiDAR Files and Index Excluded Files commands all assume the specification of YX ordering is a mistake, and instead force use of XY axis ordering. If any genuinely YX ordered LAS files are ever found, Manifold will add an option to allow YX ordering.
Why only eight files in parallel? - If we have many cores in our computer, why does the Index LiDAR Files tool only index eight of them in parallel? LiDAR files tend to be big files with the primary resource limitation being memory. Running only eight indexing operations in parallel leaves enough memory free for continuing normal operations while the tool operates in background.
Log Window - The Index LiDAR Files tool reports results into the Log Window, reporting the creation of the .mapmeta cache and the .mapindexp for each file that is indexed.
Header data cached - Processing a folder of LAS files using the Index LiDAR Files tool caches header data in .mapmeta files. .mapmeta files composed by older versions of the tool that contain no header data are accepted and are updated on the fly.
LAS over LAZ - If the same file exists in a folder in both LAS format and LAZ format, both the LAS and LAZ files will be shown as separate files to be indexed. In most cases, we would index only the LAS form, since that is faster than LAZ and also since the librarylas data port, used for building libraries from entire folders of LAS or LAZ files, will preferentially choose the LAS form for faster performance.
A good beginning - LiDAR point cloud capability in Manifold is a new feature that is just getting started. LiDAR users are encouraged to send in suggestions for any changes or additions desired. Stay tuned for many new features and improvements!
Example: Spectacular Images and Data from Web Servers
Example: An Imageserver Tutorial