NAIP images cover almost all of the United States with aerial photography in 4 bands at 1 meter or 0.6 meter resolution. We would like to download NAIP images for our areas of interest via direct download from the USGS archives on Amazon AWS. To do that we must know at least the beginning of the file name for the desired image. Unfortunately, USGS does not publish map indices with the desired names. A further complication is that the naming convention used for NAIP files is different from that used in generic USGS map indices. We can create our own indices for NAIP imagery by using the Transform pane to extract and transform the data we want from generic USGS indices for quads and quarter-quads.
This topic contains various URLs to US government servers which were accurate at the time of writing. URLs change frequently, without notice, so be ready to use your favorite search engine to find current URLs.
Download the final result, a nationwide map index for NAIP file name stems, in the NAIP_file_name_grid.mxb project published in the Examples page on the Manifold website.
See the video version of this topic in the Create USGS File Names with Transform video. The workflow in this topic takes less than 10 minutes of actual work to accomplish. See a video with tips on using the result in the Finding NAIP Imagery with Viewer video.
We will work with a map that has a Bing streets base layer. The map has one image layer that is a NAIP image showing parts of Redding, California, and two drawing layers that have been imported from a USGS file geodatabase providing map indices in the form of rectangular areas for regions covered by 3.75 minute USGS maps and for regions covered by 7.5 minute USGS maps.
Using our favorite search engine to search for "Where can I find indexes of USGS topographic maps?" on the usgs.gov website, we drill down through the National Map Download Client to find a link for a file GDB that provides map indices:
https://prd-tnm.s3.amazonaws.com/StagedProducts/MapIndices/National/GDB/MapIndices_National_GDB.zip
We can download that file, about 77 MB, unzip it to create an ESRI file geodatabase folder, and then we create a GDB data source from which we can copy the map index layers we want.
The illustration above shows the NAIP image imported from the file:
m_4012229_ne_10_060_20181009.jp2
That is located on USGS's Amazon AWS public repository. The base part, or stem of the file name that identifies the location is
m_4012229_ne
That uses a standard naming pattern, S_DDdddQQ_qq, with the following parts:
If we know the stem of the NAIP file that we want, we can visit the US government web site at:
https://catalog.data.gov/dataset?collection_package_id=54d3465c-ae77-4da1-a9d0-ed9b9ce77440
On that web page we switch the desired Order By option to Relevance and then we can search for the stem desired, for example, m_4012229_ne, and then we can click on the resulting jp2 link to download the file.
Other government websites allow us to pick NAIP files, using the stem as a guide, from lists.
Our problem is that the USGS map indices use a different naming convention to specify the locations of maps.
In the illustration above we have turned on the 3.75 minute grid. The rectangular areas in the grid have been Styled using transparent area fill color, so just the area borders appear as a rectangular grid. With the focus on the 3.75 minute grid layer, we have Alt-clicked the area covering the NAIP image to see what attributes the USGS map index layer reports for that rectangle.
As imported from the USGS file GDB, the name of the 3.75 minute grid drawing is CellGrid_3_75Minute.SHAPE, with "SHAPE" appended to show the name of the geometry field the drawing uses. For shorter names, we have renamed the drawing to exclude the "SHAPE" extension.
When we Alt-click the area, the attributes for that area object appear in the Info pane.
The main attributes used to identify the cell in which the NAIP image is located are the CELL_NAME attribute of Redding NE and the CELL_MAPCO attribute of 40122-E4. Those are a long way from the m_4012229_ne stem we would like, although we can see that part of the stem, the 40122 part, is there, as is the NE part.
USGS uses two different systems to denote the location of a quarter quadrangle (quarter quad), that is the cell that is the location of an NAIP image. Both systems are based on the idea of 1 degree by 1 degree blocks, 7.5 minute by 7.5 minute quadrangles (quads), and 3.75 minute by 3.75 minute quarter quadrangles (quarter quads). They differ primarily in how they enumerate the location of a quad cell within a larger block.
In the illustration above we have turned on the 7.5 minute quad index as well as the 3.75 minute quarter quad index. We have Alt-clicked the 7.5 minute quad area within which the NAIP image is located.
The attributes for that quad area object appear in the Info pane.
The CELL_NAME attribute for the quarter quad was Redding NE and the CELL_MAPCO attribute was 40122-E4. For the quad cell that contains that quarter quad, the CELL_NAME is Redding and the CELL_MAPCO is 40122-E4.
40122-E4 is the name of the quad within which the NAIP is located. The NAIP's quarter quad cell is simply the Northeast, or NE, quarter quad in the quad.
We can turn on the USGS map index for 1 degree by 1 degree blocks, shown above using blue lines, with the 3.75 minute quarter quad grid turned off to simplify the display. We have placed a cross symbol at the Southeast corner of the block that contains the NAIP image, with a label generated from the latitude and longitude values of the cross location. The latitude is 40 and the longitude is 122. Concatenating those, we get 40122, which is the name of the block that contains the NAIP image.
That is obviously where the 40122 text comes from that is part of the 40122-E4 name used for the quad cell containing the NAIP image. What does the E4 portion of the name mean?
USGS divides each block into eight rows and eight columns, which result in 64 subunits, the quads. Quads are identified with a letter and number combination starting from the lower right corner, the Southeast corner (the location of which is used to name the block). Each row is identified by a letter, from A to H, and each column is identified by a number, from 1 to 8.
The illustration above shows the letter and number combinations for the quad cells in each corner of the block, as well as the quad cells starting from the right, Eastern border in the row that contains the NAIP image. Using the letter/number, row/column nomenclature, the NAIP image is located in the E4 quad. Concatenating that to the block name, and we get the USGS quad identifier of 40122-E4 for the quad that contains the NAIP image.
NAIP file names use a slightly different system, where the quad identifier for the quad containing the NAIP image is 4012229. That is the name of the block, 40122, together with a number that represents the position of the quad within the block, 29. How does the number 29 identify the position of the quad within the block?
This is an alternate system for identifying quads within a block that starts numbering quad cells from the upper left, Northwest, corner. Each cell is numbered from left to right, and then by row top to bottom. Cells in the first row are numbered from 01 to 08, then the next row starts with 09, and so on until the last row starts with cell number 57 in the lower left and ends with cell number 64 in the lower right.
If we count cells we can see that the row in which the NAIP image is located starts with cell number 25 and then continues in left to right order with 26, 27, 28, and 29 for the quad cell containing the NAIP image. Concatenating the number 29 to the block number of 40122 we get the NAIP file name identifier of 4012229 for the quad cell that contains the NAIP image.
We can show both numbering systems together to see how they are related, to see how 40122-E4 in one system refers to the same quad cell as 4012229 in the other system.
Note that the second system, using numbers instead of a letter and number combination, is not truly a numeric system, because it depends on adding a leading 0 to numbers less than 10. For example, the quad called 40122-H8 in the letter and number system is 4012201, and not 401221, in the other system. When making conversions, we have to pad quad cell numbers less than 10 with a leading zero character.
Note that in both systems, the quarter quad location of the NAIP image is given by either a NE or ne. It is the Northeast quarter quad, or quadrant, within the quad. In the case of the letter and number system, the NE is part of the text name of the quarter quad, Redding NE, and in the NAIP file name system it is part of the file name, _ne added on to the name of the quad to form 4012201_ne.
The heart of the matter is converting a code like E4 into a number like 29 for each quad cell. If we replace the letters A through H with the numbers 0 through 7, and we call those numbers Y, and if we refer to the numeric part as X, the basic expression for the numeric code for each cell is 65 - (Y * 8 + X).
We can transform the letter and number system into NAIP file names using a sequence of Transform commands that implement the above expression, plus doing necessary conversions and padding. The sequence as shown below is lengthy because we do everything step by step, using previews at almost every step to check what we are doing before putting a command into action, and saving intermediate results to new fields in the table.
We will work with the 3.75 minute grid drawing and table, creating a series of intermediate results which ultimately we will concatenate together to form the correct file name stem for each quarter quad cell. We begin by extracting the two letter quadrant identifier.
We start with a display as seen above, the map being simplified to only a few layers. We right click on the 3.75 minute layer tab and choose Open Table to open the drawing's table.
To reduce visual clutter, in the illustration above we have used the Layers pane to hide fields we will not use. Our first task is to extract the two letter quadrant from each of the cell names.
With the focus on the table window, in the Transform pane we choose the CELL_NAME field. We double-click the Reduce template to launch it.
In the Reduce template, we choose end for the Reduce to option. We enter 2 for the Length.
For the Result destination we choose New Field, and we enter Quadrant as the name of the new field.
Press Preview.
The preview shows what the Reduce template will create. In the Transform pane, press Transform.
A new text, nvarchar, field called Quadrant is added to the table, and is populated with the values the preview showed. However, the quadrant values are in upper case, and we want them in lower case. That is easy to do.
In the Transform pane, we press the Up button to move up one level, so we can choose a different template.
We choose the Quadrant field, and then we double-click the Case template to launch it.
We choose lower as the Case option. We leave the Result destination as the default Same Field, meaning we will transform the Quadrant field values in place.
Press Preview.
A preview column appears, showing the change of case that we propose to apply to the Quadrant field. A dot appears in the Quadrant column head (if it is wide enough to show the dot) to indicate that field will be transformed in place.
We like what we see, so we press Transform.
Immediately, the change is applied, with quadrant identifiers transformed into lower case.
We have just extracted the quadrant for each record in the table. Although illustrations in this topic show only a small part of each table, the Transform operates on the entire table. If our 3.75 drawing and table cover the entire United States, we have extracted the quadrant for over 264,000 records, the entire US. Super!
We next will extract the quad identifier, that is the 34166 from 34116-G6 and the 40122 from 40122-E4.
With the focus on the open table window, we choose the CELL_MAPCO field. We double-click the Reduce template to launch it.
In the Reduce template, we choose start as the Reduce to option. We enter 5 as the Length.
For the Result destination, we choose New Field and we enter Cell as the name of the new field.
Press Preview.
The preview shows that reducing the CELL_MAPCO field to the first five characters will extract the quad identifier we seek.
Press Transform.
A new text, nvarchar, field called Cell is added, populated with the quad identifier.
Next, we will extract the two character letter and number identifier for the quad.
The Transform pane is still loaded with the Reduce template and the CELL_MAPCO field. We switch the Reduce to option to end, and we enter 2 for the Length.
In the Result destination, we choose New Field and enter Letter for the name of the new field.
Press Preview.
The preview shows how the two character identifier will be extracted from the last two characters.
Press Transform.
A new text field called Letter is created and populated with the two character identifiers.
We can change to another text field within the same Reduce template, so we choose Letter as the Field.
We leave the Reduce to option at the same end setting previously used. We change the Length to 1.
For the Result destination we choose New Field and enter Number for the new field.
Press Preview.
That extracts the numeric part of the Letter field.
Press Transform.
A new text field called Number appears, populated with the character giving the numeric part of the letter and number identifier.
We now will reduce the Letter field to only the first character.
We change the Reduce to option to start, and we change the Result destination to Same Field, to alter the Letter field in place.
Press Preview.
The preview shows how reducing the Letter field to the first character from the start will leave only the letter part. In the illustration above, we have slightly widened the Letter column so the dot can appear that shows the column will be transformed in place.
Press Transform.
The Letter column is transformed to be reduced to only the first character, the letter part of the identifier.
When we extracted the letter part of the identifier to create the Letter column, we extracted the text characters A through H. We will now use a Transform Expression template to create a number from 0 to 7 for each letter that we will use in the arithmetic of converting to numeric identifiers for each quad cell.
Using the Schema dialog, we add a field called LetterNum that is type int32 to the table, as seen above. The use of the Schema dialog is not illustrated: see the Schema topic for the simple procedure to add a field.
With the focus on the open table, we choose the LetterNum field. Double-click on the Expression template to launch it.
In the Expression template, click the Edit Expression button to launch the expression builder dialog.
In the Expression builder we enter the following expression.
CASE [Letter]
WHEN 'A' THEN 0
WHEN 'B' THEN 1
WHEN 'C' THEN 2
WHEN 'D' THEN 3
WHEN 'E' THEN 4
WHEN 'F' THEN 5
WHEN 'G' THEN 6
ELSE 7
END
This is a short and simple conversion, so we simply use CASE with no need for sophisticated computations. We are taking it on faith that there will be no other letters but A-H in the data, but if there are exceptions the cell in which they occur will likely not be useful anyway. Press OK to return to the template.
We leave the Result destination at the default Same Field.
Press Preview.
A quick look tells us that our CASE expression does the job. We press Transform.
That populates the LetterNum field with the translation into numbers of the letters A through H in the Letter field.
We now have all the parts we need to write an expression that creates a numeric quad grid identifier, to convert identifiers like E4 into numbers like 29.
The Transform pane is still loaded with the Expression template that creates a numeric field, so we can use that to write a new expression that we can save into a different numeric field.
Press Edit Expression to launch the expression builder dialog again. We delete the old expression in the dialog so we can write a new expression.
Recall that the basic expression we came up with near the beginning of this example was 65 - (Y * 8 + X). In the table we have created, the LetterNum field is the Y number in that expression, and the Number field is the X number in the expression. But the Number field is a text field, so we will CAST it on the fly to convert it from text data type to numeric data type.
The new expression we write is:
65 - ([LetterNum] * 8 + CAST([Number] AS INT32))
Order of precedence means that LetterNum will be multiplied by 8 before the addition, so we do not need to use parentheses to force that order of precedence.
Press OK to get back to the template.
For the Result destination, we choose New Field and then enter GridLoc as the name of the new field that will be created.
For the Result type we choose int32, to save it as an integer field and not the default float64 floating point data type.
Press Preview.
The previewed results of the expression look right, so we press Transform.
The template creates a new numeric field called GridLoc in the table, and populates it with the results of the expression.
We have just created a numeric identifier in the GridLoc field. However, we need this in text form for two reasons. First, we must concatenate this number with text to form the quad cell identifier, and second, we must pad GridLoc numbers from 1 to 9 with a leading zero, to form 01 to 09 text equivalents.
There are various ways of creating a text form from numbers. One way is to use the Schema dialog to create a text field, and then use the Expression template to do a CAST (data type conversion) of the numeric field that is saved into the text field. That is an easy approach that beginners often use.
But if we already have a text field in the table, a shortcut way is to launch the Expression template using that text field, to tell it we want a text result, and then save the result of the CAST into a different field. This avoids an additional step of using the Schema dialog to add a new field to the table, but instead using the Transform pane to do that as part of saving the Result.
In the Transform pane, we press the Up button to move up one level, so we can choose a different data type field.
We choose the Letter field, and then we double-click the Expression template to launch it.
The Expression template now knows we want the result of the expression to be a text field. We press Edit Expression and in the expression builder dialog (not illustrated) we enter the expression:
CAST([GridLoc] as NVARCHAR)
Press OK to return to the template.
Back in the Expression template, we choose New Field as the Result destination, and we enter GridLocText as the name of the new field.
Press Preview.
The preview shows what appear to be the same values as the GridLoc field. They look the same because 11 as a numeric data type is the same as 11 in text form.
Press Transform.
A new text field called GridLocText appears in the table, populated with the text equivalent of the numbers in the GridLoc field.
We scroll the table down a few rows, to where we can see GridLocText values that are single digits. We need to pad those with leading zero characters.
Lucky for us, Manifold has a transform template for that.
In the Transform pane, we press the Up button to move up one level, so we can choose a different data type field.
With the focus on the table window, we choose the new GridLocText field and we double-click the Pad template to launch it.
In the Pad template, we choose start as the Pad option, with 0 for the Repeat value and 2 for the Length. We are telling the system to repeatedly enter the 0 character at the start of each text for as many times as is necessary for the total length of the text to be 2 characters.
For the Result destination, we leave the default Same Field choice in place, to transform the field in place.
Press Preview.
Like magic, the preview shows each of the single digit values padded with a leading zero character. The dot in the GridLocText column head indicates that field will be transformed by what we see in the preview.
Press Transform.
Super! We now have a padded, text form of the quad grid location. We now have all of the pieces we need to assemble NAIP filename stems, like m_4012229_ne.
In the Transform pane, we press the Up button to move up one level, so we can choose a different template.
With the focus on the open table window, we choose the Cell field. We double-click the Concatenate template to launch it.
In the Concatenate template, we choose start as the Add at option. We enter m_ as the Value option.
For the Result destination, we choose New Field and enter Stem as the name of the new field.
Press Preview.
We drag the preview column next to the Cell column in the table, so we can more easily compare the preview. We see that it indeed takes the Cell text and prepends m_ to the text. The m_ prefix indicates an NAIP image acquired by a four-band sensor, as used for modern NAIP imagery, so we want that as part of our NAIP file name stem.
Press Transform.
A new text field called Stem appears, loaded with the results as shown in the preview.
We will now concatenate the quad grid location text to the stem.
The Concatenate template is still loaded in the Transform pane, so we change the Field to Stem and we change the Add at option to end.
For the Value to be appended, we choose from the pull down menu the GridLocText field.
For the Result destination, we choose Same Field, to transform the Stem field in place.
Press Preview.
From the preview, we see that the GridLocText texts are correctly appended to the Stem texts.
Press Transform.
The template updates the Stem field by appending the GridLocText field values.
We have just one more concatenation to do, and then we are done. We must append the Quadrant for each row onto the end of the Stem text.
The Transform pane is still loaded with the last Concatenate we did. We will again add to the end of the Stem field.
For the Value field we choose Quadrant from the pull-down menu.
For the Separator, we enter an underscore _ character. The Separator text will automatically be inserted in between the Stem field and the Value text that will be appended to the end.
Press Preview.
The preview shows the result is exactly what we want.
Press Transform.
The Stem field is transformed as shown in the preview.
We have now accomplished our task. Each quad identifier using letter/number format and separate field giving the quadrant as part of the name of the quad, has been transformed into the equivalent NAIP file name stem form.
We can create labels from the Stem field, and add the labels layer to our map:
Seen above, labels show our procedure we used is correct. Now is a good time to save our project, if we have not yet (!) done so. Our 3.75 map index drawing layer now has an additional Stem field that we can use to find the NAIP file we want.
For example, we can Alt-click the quarter quad box next to the one in which our NAIP image is loaded, to get the file name stem for the adjacent NAIP image.
The attributes for the Alt-clicked rectangle are displayed in the Info pane. We right-click onto the Stem value and choose Copy to copy it to the clipboard.
There are many ways to find online the NAIP file we want, by using the file name stem to search. One of the easiest is to visit the data.gov site for searching the NAIP imagery collection, at:
https://catalog.data.gov/dataset?collection_package_id=54d3465c-ae77-4da1-a9d0-ed9b9ce77440
The above link provides a page that is useful for searching the NAIP imagery collection:
The default Order by setting is the Popular option, not so useful for finding a file by name, so we switch that to Relevance (as shown below with a magenta arrow).
We right-click into the search box and Paste the text we copied from the Info pane. Press Enter, and the page returns over 218,000 results, but mercifully, the file we want is the first one.
Clicking the image/jp2 button will launch a download of a .jp2 file containing the NAIP image file we want. The direct link is:
Other links returned when we search for the file name stem m_4012230_nw can be the same NAIP image location, but acquired in different years and different resolutions.
Intrepid and street smart use of search engines will reveal a variety of sites for downloading NAIP files. For example, we can visit the page at:
https://thor-f5.er.usgs.gov/ngtoc/metadata/waf/orthoimagery/naip/
The above page lists collections of NAIP imagery by year and by state. Drilling into a year and state, such as 2016 for California, we can reach pages like:
https://thor-f5.er.usgs.gov/ngtoc/metadata/waf/orthoimagery/naip/ca_2016/
The above is a list of metadata files. To find the file of interest for the NAIP image we want, we use Ctrl-F in our browser to Find the file name stem, such as m_4012230_nw in the page:
We click on the link to read the metadata, using Ctrl-F again to find m_4012230_nw in the text. The <networkr> entry gives the full URL to the Amazon AWS .jp2 file for that NAIP image.
The main advantage of using a site like the one above is that it provides many adjacent NAIP files near each other in the same list.
Data.gov links - Data.gov can be challenging to use, so if the link cited above for search the NAIP imagery collection no longer works we might have to spend some time finding the new NAIP imagery collection search page.
Why not Earth Explorer? - Why not download NAIP files from the US Government's free earthexplorer.usgs.gov web site? The main reason is that Earth Explorer can be a huge pain to use, and it is extremely slow. It requires an iterative process of drawing map shapes and then seeing what files pop up, checking their footprints to see if they are desired, and so on. It is much easier to consult an index grid that overlays an area of interest to get the specific files desired and to download them directly from the high speed Amazon AWS archives. Keep in mind that creation of an index giving NAIP file name stems is a one time task, with that same layer being useful for years.
Field name differences - The illustrations in this topic were created using the collection of map index shapefiles published by USGS for individual states, available at:
https://prd-tnm.s3.amazonaws.com/index.html?prefix=StagedProducts/MapIndices/Shape/
Field names in the shapefiles are slightly different than field names in the GDB, for example, using CELL_MAPCO instead of CELL_MAPCODE
Yet another useful site - If we do not mind browsing through a list of all of the NAIP files in a single block, we can get a list of the AWS archives using:
https://prd-tnm.s3.amazonaws.com/index.html?prefix=StagedProducts/NAIP/
Click into the state and year of interest, and then use the Cell number in the example above to drill into a block to see all of the .jp2 NAIP files for that block. Tools like Uget can download all of them at once.
Download the final result, a nationwide map index for NAIP file name stems, in the NAIP_file_name_grid.mxb project published in the Examples page on the Manifold website.
Create USGS File Names with Transform
Finding NAIP Imagery with Viewer
Example: Display an NAIP Four Band Image as Color Infrared (CIR) - How to use the Style pane for images to re-assign channels in a four band NAIP image to produce a Color Infrared (CIR) image display.
SQL Example: Create NDVI Displays - How to create a query that creates an NDVI display from a four-band NAIP image, with tips and tricks on how to copy and paste existing information to get the result we want.
Example: Two Drawings from the Same Table - Take a table with a geom field that is visualized by a drawing. Add a second geom field to the table and create an rtree index on that field so it can be visualized by a drawing. Copy the first drawing, paste it and adjust the pasted copy so it uses the second geom field.
Example: Construct JSON String using Select and Transform - Use the Select pane and the Transform pane to manually construct a JSON string using values from other fields in a table. Shows how we can manipulate text to build desired contents in a field.
Example: Copy one Column into Another Column with Transform - How to use the Transform pane to copy the contents of one column in a table into another column, but only for selected records. Uses the Products table from the Nwind example data set.
Example: Use a Transform Expression to Create Buffers in a Drawing - Use the Expression template in the Transform pane to create three different sizes of buffers for different lines in a drawing and then automatically create a query which does the same thing. Includes examples of saving results to a new component and also the Edit Query button.
Example: Clip Areas with a Transform Expression - Use the Expression template in the Transform pane to clip areas in a drawing to fit within horizontal bounds. Includes examples of saving results to a new component and also the Edit Query button.
Example: Simplify Lines with a Transform Expression - Use the Expression template in the Transform pane to make lines smoother in a drawing so that longer lines are smoothed more. Includes examples of using the Edit Query button to show how different queries are created automatically depending on if we want to update a field or to add a new component.
Example: Transform Templates, Expressions and Queries - We learn to use a function by clicking on a template in the Transform pane, seeing what it does in a preview, looking at the query Manifold creates and then trying out the function in the Expression tab.