The librarylas dataport creates a data source from all the LAS or LAZ files in a folder and sub-folders at once, presenting the combined data from all of the files as if they were a single point cloud drawing. That eliminates any need to manually link to or import many files and then manually to merge the files into a single drawing. The librarylas dataport can also dynamically refresh the data source, to update the drawing to incorporate any changes in the data files, including any LAS or LAZ files that have been added to or removed from the library's folder. If the same file exists in a folder in both LAS format and LAZ format, the LAS form will be used, for faster performance.
This topic assumes we have read the LAS, LAZ LiDAR topic. Please review that topic for important background information on LiDAR, LAS and LAZ file formats, and how Manifold creates spatial indices for point clouds.
Creating a LiDAR data source using
librarylas will often involve
very many LAS or LAZ files within folder and subfolders, so librarylas assumes
users will have used the Tools
- Index LiDAR Files command
to build indices in advance for the LiDAR files in the folders involved.
If indices have not been built in advance, librarylas
provides a way to build indices for any LiDAR files in the subject folders
without indices, but it makes more sense to build such indices in advance.
In the main menu choose File - Create - New Data Source, and then More...
In the New Data Source dialog provide a Name for the new data source if the default is not desired.
Choose Folder: librarylas as the Type of data source.
Click the [...] browse button to launch the Select Folder dialog.
In the Select Folder dialog, Click the desired folder to highlight it, and then press Select. That will load the folder as the Source.
Set option boxes that appear as desired, for example, unchecking the Save cached data between sessions.
Press Create Data Source.
A new data source appears in the project. Expand it to see the drawing and table it contains.
If the new data source has a cylinder icon with a + sub-icon, that means some of the LiDAR files have not yet been indexed.
If so, right-click onto the name of the data source and choose Index Excluded Files.
If individual LAS or LAZ files within the folder have more or fewer attribute fields, the composite table will use the greater number of fields. Point records from files that have fewer fields will have NULL values for fields not in those files.
The LAS library dataport exposes data in built-in and custom LAS headers from individual files as Info and InfoVLR tables. Both tables contain an additional field for the file path.
Manifold will read any coordinate system information specified by LAS or LAZ files. In addition, Manifold also will get projection (coordinate system) information from any sidecar PRJ or PAM XML files that accompany the LAS or LAZ files. If an accompanying file specifies a different coordinate system than the LAS file specifies, the sidecar file wins, since sidecar PRJ or PAM XML files are often provided to override, that is to fix, erroneous coordinate information specified within the LAS.
Individual LAS or LAZ files within the library's folder might use different coordinate systems. If individual LAS or LAZ files use different coordinate systems, the librarylas dataport picks one of the coordinate systems at random and dynamically projects data from all files into that coordinate system.
Linking a LAS or LAZ file exposes coordinate system shifts and scales for X, Y, and Z fields as properties in the linked drawing's table. Shifts and scales to the geometry field. X, Y, and Z fields are float64 data types and report scaled coordinate values. Exporting to LAS or LAZ uses this data to produce a file with the exact same shifts and scales as the original.
The LAS library dataport automatically computes shifts and scales for the entire data set. If the files making up the library are in the same coordinate system and use the same scale, the library will use that scale, provided the scale is large enough to fit all data into numeric range of 32-bit integers.
Important: The above features were introduced
in build 173.5 and updated
in build 173.6. When
linking to LAS or LAZ files previously used in Manifold builds before
173.6, delete any .mapmeta file
in the same folder that were created by any builds before 173.6.
That will prevent the old .mapmeta file
from limiting the properties created to only those used in builds before
173.6.
Exporting a linked LAS Library drawing or the drawing's table works the same way as exporting a linked LAS or LAZ file. See the LAS, LAZ LiDAR topic for details.
Use the Tools - Index LiDAR Files command to create spatial point cloud indices in advance for LAS or LAZ files in the folder and all sub-folders that will be used for the librarylas data source. If we fail to do that, we will encounter two inconveniences:
Despite a special icon being used for the data source to indicate some indices are missing, we may not notice some data is not displayed, for lack of an index, and that we need to launch the Index Excluded Files command to provide missing indices.
If we do notice some indices are missing, when we launch the Index Excluded Files command we will have to wait for the indices to be built. That could be a long time if many of the LiDAR files in the subject folder do no not have indices.
Thinking ahead can save waiting around: when we first download some LAS or LAZ files, launch a Manifold session and run the Tools - Index LiDAR Files command to create indices for the downloaded files. Leave it running in parallel while we continue doing work in other Manifold sessions at the same time, and for bigger collections of files, leave it running overnight. When indices have already been created for the LAS or LAZ files we intend to use, when we link them into our projects they will be ready for use very quickly with no need to wait around while indices are created.
If we have not created
indices in advance, when we create a librarylas
data source, when we expand the data source to see what it contains the
data source will be marked with a special cylinder icon that includes
a + sub-icon
to indicate that indices need to be added.
The example below shows the worst case scenario, where we have neglected to build indices in advance using the Index LiDAR Files command. We will build those indices using the Index Excluded Files command.
We will create a data source using a folder of LAS files that provide LiDAR data for the city of Portland, Maine. Files were downloaded from NOAA servers.
In this example we use LAS format files, which we created from the original LAZ files that were downloaded by using the LASzip.exe utility. librarylas is happy to use either LAS or LAZ format, but for the reasons given in the LAS, LAZ LiDAR topic, LAS format is significantly faster when many or large LiDAR files are used.
Some of the files have been used previously in Manifold and thus have .mapindexp files containing a pre-built point cloud spatial index, but many of the files do not have a point cloud index. For the sake of this example, we have not in advance run the Tools - Index LiDAR Files command to create indices for the LiDAR files we intend to use.
We choose File - Create - New Data Source, and then we choose More...
We choose a dataport Type of Folder: librarylas. We provide a name, Portland LiDAR, and we uncheck the Save cached data between sessions box. Caching data between sessions results in large project file sizes, so we uncheck that box. We press the browse button to navigate to the file folder which will be our Source.
In the Select Folder dialog, we navigate to the folder which holds our collection of LAS files. We click on the folder name and then press Select.
That loads the path to the folder that holds our collection of LAS files. The Recurse into subfolders box is checked by default, so if we had any subfolders that contained more files, those too would be acquired.
We press Create Data Source.
A new data source appears in the Project pane. When we expand the data source, a new drawing and table appear in the data source, with names based on the name of the folder we chose, in this case Portland LiDAR Drawing for the drawing and Portland LiDAR for the table.
The data source also shows a +
sub-icon in the cylinder icon, to show that some files in the target folder
for the library were excluded for not having indices.
We create a map using Bing streets as a base layer. We drag and drop the Portland LiDAR Drawing into the map.
When we drop the Portland LiDAR Drawing into the map, points start filling in, with a blue underline in the drawing's tab indicating it has not yet finished rendering.
Only those points for LiDAR files that have a point cloud index in an .mapindexp accessory file will appear. LiDAR files without an index will be excluded. The librarylas dataport reports in the Log Window the total number of LAS/LAZ files included in the library (those with indices) and the total number of files excluded from the library (those without indices).
To index any LiDAR files without indices, we have two options.
We can right-click the data source in the project pane (in this case, the Portland LiDAR data source with database cylinder icon) and choose Index Excluded Files.
That will launch the Index Excluded Files command to automatically index any files in the LAS library that do not have indices. The indexing process will run in parallel, indexing eight files at a time in 64-bit mode and two files at a time in 32-bit mode. When the process is finished, the points for formerly excluded LiDAR files in the library folder will appear in the map.
The Index Excluded Files command is convenient because it will automatically index all files without indices that participate in the LAS library data source we have created.
Alternatively, we can use the Tools - Index LiDAR Files command. In the Index LiDAR Files dialog, choose the folder containing the LiDAR files to be indexed.
Ctrl-click any files that report no index (all the files in the illustration above have indices), and then press Index.
The Index LiDAR Files command at will start an indexing process that runs in parallel, indexing eight files at a time in 64-bit mode and two files at a time in 32-bit mode. When the process is finished, the points for formerly excluded LiDAR files in the library folder will appear in the map.
The Index
Excluded Files command as well as the Index
LiDAR Files command both work in the free
Viewerl,
to build point cloud spatial index files for LAS or LAZ files that do
not yet have a point cloud spatial index. Super!
If indices are required for the 39 LAS files in this example, totalling approximately 13 gigabytes in size and collectively containing approximately 463 million points, the process of building index files for all 39 files requires about eight minutes on a newer computer using an SSD drive for storage. An older machine using a hard disk will take longer, perhaps ten or twelve minutes. Using LAZ files will run slower due to the need to first decompress the LAZ archives.
Avoid waiting: Use the Tools
- Index LiDAR Files command
to build indices for LiDAR files in advance of need. If all
of the LiDAR files in the folder and sub-folders used for a librarylas data source have indices, there is
no need to run the Index LiDAR
Files command.
The librarylas dataport automatically detects the coordinate system used by the LAS files, so the drawing appears with correct georegistration, automatically being reprojected on the fly into the Pseudo Mercator projection used by the map.
Manifold automatically applies thinning to the points displayed, since attempting to show all 463 million points would result in a black mass. The illustration above shows the drawing when it has finished rendering, with point size set to a 3 points and using .5 point stroke thickness.
We can see how the composite drawing created by the librarylas dataport relates to the 39 individual LAS files which provide data by overlaying a drawing that shows the footprints, that is, the outlines, of the contents of the individual LAS files. The NOAA website helpfully provides a shapefile for the data set that shows the outlines of the region covered by each individual LAS file. The librarylas dataport has taken points from all of the LAS files and has presented them as a single drawing, as if we had linked all the files individually and then merged them to create a single drawing.
If we zoom further into the view, Manifold will populate the display with more and more points.
Popping open the Portland LiDAR table created by the dataport, we can see it has many attribute fields in addition to the mfd_id and Geom fields synthesized by Manifold.
To see the full names of the fields we can take a look at the Layers pane. We can use any of these fields in SQL to extract data as desired from the table based on various attributes.
Taking a look at the file folder we used in this example, we see that for each .las file, Manifold has created a .mapindexp file containing the point cloud spatial index and a .mapmeta file containing metadata in JSON form, such as coordinate system information. These files will be used in any future session of Manifold that uses any of the .las files, eliminating the time spent during their first use to build the spatial index.
We can update the drawing within a librarylas data source by right-clicking on the data source name and choosing Refresh.
Refresh will update the drawing to incorporate any changes in the LAS or LAZ files within the folder hierarchy specified for the library, including any LAS or LAZ files added to the folder or deleted from the folder. Refreshing a library may change the schema of the drawing's table due to changes in the LAS or LAZ files in the folder. Different versions of LAS files expose different fields, and LAS files may contain custom fields. Refreshing a LAS library updates the coordinate system, along with any shifts and scales defined in the coordinate system.
We can select desired data from a librarylas data source the same as from a LAS or LAZ data source. See the LAS, LAZ LiDAR topic for an example.
Regrettably, it is routine to encounter LAS/LAZ files that wrongly specify coordinate system axis ordering of YX when in fact the data uses XY axis order. Such errors are a classic case of wrongly specified coordinate systems discussed in the That YX Thing essay.
To date, no LAS/LAZ files have been encountered that specify YX axis ordering and which in truth use YX axis ordering. Therefore, the Manifold LAS, LAZ LiDAR and librarylas dataports as well as the Index LiDAR Files and Index Excluded Files commands all assume the specification of YX ordering is a mistake, and instead force use of XY axis ordering. If any genuinely YX ordered LAS files are ever found, Manifold will add an option to allow YX ordering.
Log Window - The librarylas dataport reports in the Log Window the total number of LAS/LAZ files included in the library (those with indices) and the total number of files excluded from the library (those without indices).
Dynamic file management - The dataport exposes a single composite drawing for all files, but it manages access to individual files dynamically for maximum performance. For example, a LAS library will not open all files on startup. Instead, it will only open constituent files as their data is required. For example, if we zoom into a map window so that the window view only shows part of the region covered by the LAS library drawing, and then we drag and drop the LAS library drawing into the map, only those constituent files that are required to cover the map window will be opened for reading to fill the display, potentially saving a lot of time. Zooming to fit for the entire library will eventually open all or most files, depending on how small their extents are. Likewise, queries against the entire drawing will use all files, since all of the data will have to be read.
Attribute fields - If individual LAS or LAZ files within the folder have more or fewer attribute fields, the composite table has the greater number of fields. Point records from files that have fewer fields will have NULL values for fields not in those files.
Reporting - The dataport reports the file source ID even if it is zero. Attribute fields report the file path for each point.
LAS over LAZ - If the same file exists in a folder in both LAS format and LAZ format, the LAS form will be used, for faster performance.
A good beginning - LiDAR point cloud capability in Manifold is a new feature that is just getting started. LiDAR users are encouraged to send in suggestions for any changes or additions desired. Stay tuned for many new features and improvements!
Example: Spectacular Images and Data from Web Servers
Example: An Imageserver Tutorial
Example: Vector to Raster using Kriging