OR/15/057 Data processing

From Earthwise
Jump to: navigation, search
Jones, L D. 2015. Ground-based geomatic surveys at the BGS - a manual for basic data collection & processing (2015). British Geological Survey Internal Report, OR/15/057.

Workflow procedure

Superceeded workflow

There are currently no Quality Assured documented standard procedures for field data acquisition, data processing/visualisation, or the storage/retrieval of raw data, photographs and processed 3D models. The current work-flow is fragmented, complex and time consuming (Figure 13). Scan data and GNSS positioning data collected in the field varies with respect to accuracy and precision, depending on the individual operator collecting it and also on the type of survey being carried out. Back in the office the scans are oriented using the GNSS data, attributed with their RGB colour and correctly aligned. The data are exported from RiProfile (or RiScan if VZ data) and imported into one of nine software packages in order to grid, map, model or visualise it — no single package is able to carry out all the tasks required at present.

Figure 13    Work-flow.

Current workflow

The ‘new’ work-flow is much more simplified and less time consuming than the superseded work-flow (Figure 14). Scan data is now collected in the field in a much more rigorous manner, with users aware that resolution and accuracy are of equal importance. GNSS data is also collected in a more measured and precise manner, with users aware of issues that might arise and the ways in which to deal with them. The scans are geo-rectified using the corresponding GNSS data, properly aligned using multi-station adjustment algorithms, and attributed with RGB and Amplitude (intensity) values. This work is now carried out entirely in RiscanPro (for the Riegl scanners) or Scene (for the Faro scanner) and the data are exported as ASCII files. These data are then imported into one of six software packages in order to grid, map or model the data and one of two software packages in order to visualise it.

Figure 14    Current work-flow.

Data issues

Modern laser scanners have significantly increased the level of resolution, accuracy and speed of data acquisition. However, the extremely large data files produced by the latest terrestrial LiDAR surveys (typically 40–50 GB raw and 80–100 GB processed) are making them difficult to deal with. Although the processing power of a typical desktop PC has increased significantly over the past few years the modelling software’s ability to deal with these increasingly large datasets has not.

A further major issue associated with the large datasets produced by terrestrial LiDAR surveys is the difficulty of storing and archiving the data. This has recently been addressed with the implementation of a corporate TLS Data Plan:

  • TLS data collected for many projects in BGS by trained TLS users
  • Raw TLS data stored on W:\Teams\SGR\Geomatics\Data under Project folder
  • Data checked, geo-referenced and trimmed (if required) in this folder
  • Copy of Raw and Processed data kept on this drive
  • Data exported as ASCII point clouds to S:\Geomatics\Terrestrial where it is stored under Project folder (same name as W:drive)
  • Data cleaned and filtered, multiple files combined and .dat files created for GeoVisionary
  • Completed processed data now available for access by all BGS users, not just commissioning project
Figure 15    Corporate Data Work-Flow.

Processing walk-through

Once the survey is completed the data needs downloading from the various pieces of equipment; GNSS positional data from the Compact Flash cards in the Leica SmartRover Controller and Viva Controller and SD cards in the Viva Antenna. LiDAR data from the HDD on the Toughbook associated with the LPM scanners and from the HDD on the Getac associated with the VZ scanner.

Leica geo office

The GNSS positional data is stored on Compact Flash cards located inside the Smart Rover and Viva Controllers, and on SD cards located inside the Viva Antenna. The data can easily be copied from the ‘DBX' folder on the card to a PC using a card reader. Start LGO.

  • Choose 'Projects'
  • Right-Click <Project> folder then <New>
OR15057screenshot1 7.2.1.jpg

Type in the name of the survey and click <Coordinates> Tab. Change the Coordinate System to OSGB36 <OK>

OR15057screenshot2 7.2.1.jpg
  • <Import> <Raw Data>
  • Browse to ‘DBX’ folder and select the data and ‘Assign’ it to the required survey
OR15057screenshot3 7.2.1.jpg
  • Select ‘Points’ tab to view imported data
  • In View change Coordinate System to ‘Local' and Coordinate Type to ‘Grid’. <Tools> <Compute Geoid Separations>
OR15057screenshot4 7.2.1.jpg

Note: Processing RINEX data is not covered in this manual.

  • If you are dealing with RINEX data you will need the Dongle (No. 27) in order to import and post-process it in LGO
  • <Export> <ASCII Data> Choose location to save to and ‘Save As’ *.csv format
OR15057screenshot5 7.2.1.jpg

This can now be ‘cleaned-up’ in Excel and saved as a (comma-delimited) *.csv file.

Note: ‘Orthometric height’ is correct value to use, not ‘Ellipsoid height’.

Open the *.csv file with Notepad and save it with no headings in the form {Name, X, Y, Z} and save as a *.uda file. It is useful to also create a ‘space delimited' *.txt file in the form {Name, X, Y, H}, where H is Z + Scanner Height — This ‘Height" file will be used in Polyworks later.

OR15057screenshot6 7.2.1.jpg

Riegl RiProfile

RiProfile is the LPM scanner operating and visualisation software. It allows the user to run the laser scanner and also to orient, join and view the scans. The data is stored on the Panasonic Toughbook laptop in a folder named after the survey and can be opened using the 'Project.rsp' file.

  • The first step, once the project is loaded, is to check that all the scans are there, and appropriately named
  • Next, create a coloured point cloud of the images. To do this they must firstly be undistorted. Open ‘SCANPOSIMAGES’, highlight all images and select ‘Undistort’
  • ‘Right-click’ on a Scan and select ‘Colour From Images’. Change to ‘Undistorted Images’ and highlight all images (for Scan 1 only). <OK>
  • To view the data properly in RiProfile use ‘True Colour', ‘Linearscaled', ‘Type = Amplitude', ‘Sub = Histogram'
  • In order to view all the scans together then it is a good idea to filter the data. <Registration> <Multi Station Adjustment> <Prepare Data>
OR15057screenshot1 7.2.2.jpg
  • Select all the scans required and then run an 'Octree' filter with the following settings:
OR15057screenshot2 7.2.2.jpg
  • Alternatively you can run a ‘2.5D’ filter, which may help in finding the ground level on a vegetated survey, or an ‘All' filter, which keeps more detail.

Note: These filters will create Polydata in the OBJECTS folder.

  • In order to correctly align the scans to National Grid, and to each other, they must be oriented. Right-Click on ‘SOP’ and choose ‘Backsighting Orientation’. Put in the scanner position (use the *.uda file) and instrument height (recorded in field notebook). Click <Next> and enter the backsight target position
OR15057screenshot3 7.2.2.jpg
  • And Alignment
OR15057screenshot4 7.2.2.jpg
  • Click <Next>. Select <Set SOP>. SOP icon will go blue and star will appear next to POP. Close ‘Backsighting Orientation’
  • In the survey folder (*.RiProfile) create a new folder named ‘MATRICES’ and save ALL the SOP positions <Registration> <Multi SOP export> to this new folder
  • To create a Panorama image, open ‘SCANPOSIMAGES', highlight all images (for Scan 1 only). ‘Right-click’ and select ‘Create Panorama Image’ <OK>. Choose ‘Name' and ‘Location’
  • If this doesn’t work it could be because you have too many photos. Try splitting them in half and generating 2 coloured scans
  • SAVE the Project. Don't shut it down yet!

Innovmetrics polyworks (IMAlign)

Polyworks is a 3-D point cloud software platform designed for surveying applications. IMAlign allows for re-alignment and visualisation of multiple scans.

  • Run Polyworks and open ‘IMAlign’. Import the oriented RiProfile data. <File> <Import Images> <3D Digitized Datasets> <Spherical Grids> and choose RSP Project (Riegl)
OR15057screenshot1 7.2.3.jpg
  • Browse to the RiProfile folder and select all the required scans. <OK>
  • Choose Metres and Colour
  • The ‘Create Image’ window will open. Change the ‘Focus distance’ to 20 and make sure that ‘Use same values for all images’ box is ticked. <Create>
OR15057screenshot2 7.2.3.jpg
  • Group scans from the same scanner location. Highlight them, right-click <Edit> <Group>
  • Choose the ‘best’ scan and lock it. Right-click <Edit> <Lock> and ‘Ignore’ all the scans but one (other than the locked scan) Right-click <Edit> <Ignore>
  • Carry out a ‘Best-fit Alignment’ by clicking the shortcut and changing the ‘Parameters’ to the following:
OR15057screenshot3 7.2.3.jpg

Click <Start>

Check the ‘Statistics’, look at the shape of the ‘Histogram’ and run a ‘Comparison’ check on the two scans

OR15057screenshot4 7.2.3.jpg
  • Lock the second scan (keeping first locked) and carry out the method again with a third scan etc
  • If the scans are too far out (misaligned, bad backsight etc.) they can be manually adjusted using the ‘Split-view Alignment’ tool. Zoom in to the area and only use [Shift+R] to rotate around the axis of the screen and [Middle Mouse] to translate the image
OR15057screenshot5 7.2.3.jpg
  • Then carry out the ‘Best-fit Alignment’ as before
  • Once all the scans have been re-aligned the survey needs adjusting to its correct base level. To do this highlight all the digitiser positions, right-click <Edit> <Use>
OR15057screenshot6 7.2.3.jpg
  • Make sure all the scans are visible and unlocked. Highlight them, right-click <Edit> <Unlock>
  • Right-click on ‘Global reference Points’ <Create> and change method to ‘From Text File’ and browse to the ‘Height’ file (created earlier) under the ‘Names + Points’ Template (need to create this for first time of use)
OR15057screenshot7 7.2.3.jpg
  • Select all the scans and press the ‘Auto-Match’ button, choosing ‘Global reference points only’.
OR15057screenshot8 7.2.3.jpg
  • Select the scans (only require one from each Group). Right-click <Export> <Alignment Matrices> <To Text (4x4)>
OR15057screenshot9 7.2.3.jpg
  • Export these *.txt files to the ‘MATRICES’ folder under the surveys *.RiProfile project.

Back to RiProfile

  • For each scanner position, right-click the SOP <Import> and choose ‘Modify via Polyworks image file (*.*)’ and adjust each one
OR15057screenshot1 7.2.4.jpg
  • In the survey folder (*.RiProfile) create a new folder named ‘DATA’. Right-click each scan (colour or intensity) <Export> and ‘Save as' ‘xyzRGB’ (colour) or ‘xyzi’ (intensity) *.csv format to this folder. Make sure you use the GLCS coordinates
  • SAVE the Project

Innovmetrics polyworks (IMSurvey)

IMSurvey is the visualisation tool, that allows the 3D viewing of single and multiple scans and the creation and viewing of change models etc.

  • In Polyworks, open ‘IMSurvey’. Right-click ‘Data’ <Import> <ASCII Point Cloud> and bring in the scans
  • ‘Ignore’ all scans but one (usually oldest). Highlight this scan and <Select> <All Elements>
  • To create a mesh use <Edit> <Triangulate Data Points> and then re-name it
OR15057screenshot1 7.2.5.jpg
  • Initially try the XY ‘Triangulation plane’ and set the ‘Max edge length' to 5 m (ish)
OR15057screenshot2 7.2.5.jpg
  • This triangulated model needs to be converted from a ‘Data’ object into a ‘Reference’ object. The initial data object can be ‘Ignored’ (or even ‘Deleted’). ‘Select’ the triangulated model <Edit> <Data/Reference Object> <Transfer> to turn it into a ‘Reference’ object
OR15057screenshot3 7.2.5.jpg
  • To compare models (and create a change map), ‘Use’ the new ‘Data’ object (what your model is being compared to). ‘Select’ the triangulated model <Compare> <Data to Reference> <All Data Points> and change the ‘Max distance’ to 1 m (or greater), turn the ‘Max angle’ off and initially try ‘Direction’ as shortest distance
OR15057screenshot4 7.2.5.jpg
  • Use <Compare> <Error Display Options> to bring up the options box and make any changes to the display required
OR15057screenshot5 7.2.5.jpg
  • SAVE the project

Riegl RiScanPro

RiScanPro is the VZ scanner operating and visualisation software. It allows the user to run the laser scanner and also to orient, join and view the scans. The data is stored on the Getac laptop in a folder named after the survey and can be opened using the ‘Project.rsp’ file.

  • The first step, once the project is loaded, is to check that all the scans are there, and appropriately named
  • Next, right-click on TPL (GLCS), change the Files of type to ‘Any ASCII file’ and navigate to the GPS (*.uda) file created earlier. Click Open. Check the settings in the Preview window <OK>
  • Set the height for all Scanner Positions as -0.295 m (Scanner Origin Offset) and for any tiepoints as ‘say’ +2.000 m (if using Pole Targets).

Note: Height from top of battery to scanner axis is 0.233 m (needed if GPS NOT on scanner).

  • Select the ‘Calculate translation for POP’ button to set the POP (Project Orientation and Position) to the ideal value for all GPS positions in the project.
OR15057screenshot1 A.jpg
  • Select all values. Right-click and select Copy tiepoints to ‘TPL PRCS’. Close window.
  • Right-click on POP and select ‘Freeze’
  • TPL (PRCS) will automatically select points as Control points.
  • Note the change made to the Z value. Close window.
  • Next, create a coloured point cloud of the images. To do this they must firstly be undistorted. Open ‘SCANPOSIMAGES’, highlight all images and select ‘Undistort’
  • 'Right-click' on a Scan and select 'Colour From Images'. Change to 'Undistorted Images' and highlight all images (for one scan at a time). <OK>
  • To create a Panorama image, open ‘SCANPOSIMAGES’, highlight all images (for 1 scan at a time). ‘Right-click’ and select ‘Create Panorama Image’ <OK>. Choose ‘Name’ and ‘Location’

N.B. If the images have a white aura around them it could mean the camera mounting was mis-alligned and the calibration needs adjusting — See Appendix 1.

  • In order to correctly align the scans to National Grid, and to each other, they must be oriented. Right-Click on ‘SOP’ (Scanner Orientation and Position) under each scan position and choose ‘Backsighting Orientation’.
  • Click on Use GLCS tiepoint and choose the point from the list that corresponds to the Scanner position (Ignore Instrument height). <OK> Click Next.
OR15057screenshot2 B.jpg
  • Click on Use GLCS tiepoint and choose the point from the list that corresponds to the Target position. <OK> Click Next.
OR15057screenshot3 C.jpg
  • Make sure that Use inclination sensors OFFLINE box is ticked and the Panorama scan is selected. Highlight the Use tiepoint (finescanned) button and click Select tiepoint. Choose the correct tiepoint for the current Scanner position. <OK> Click Next.
OR15057screenshot4 D.jpg
  • The programme will extract the inclination values and then display the ‘Summary’ screen. Check the details are correct and click Set SOP. Click Close.
OR15057screenshot5 E.jpg
  • Repeat this process with ALL Scanner positions.
  • In the survey folder (*.RiSCAN) create a new folder named ‘MATRICES' and save ALL the SOP positions <Registration> <Multi SOP export> to this new folder. Make sure *.SOP and *.CSV are ticked and the Coord. system is GLCS. <OK>
OR15057screenshot6 F.jpg

Before the computational scan registration tool, Multi Station Adjustment, can be run it is necessary to register the scans approximately using the Coarse Registration tool. There are two ways this can be done — either by picking pairs of corresponding points in each scan, or by manually translating and rotating each scan by eye onto the fixed scan.

Picking point pairs:

  • From the Registration menu select Coarse Registration.
OR15057screenshot1 7.2.6.1.jpg
  • In the Coarse Registration dialogue box click on the ‘Open new object view’ button for View A. An empty Object view window opens. Drag & drop the ‘fixed’ scan from the Project manager window into the new Object view window. A Select view type… dialogue box opens. Select True color, linear scaled option and click OK.
OR15057screenshot2 7.2.6.1.jpg
  • Repeat for View B and drag & drop the first scan that you want to coarsely register with the fixed scan. Click on the Arrange views button to display both views side by side.
OR15057screenshot3 7.2.6.1.jpg
  • View the data in both windows from above by clicking on the birds-eye view button OR15057birdseyeview1.jpg. Rotate the scan data in View B using SHIFT + Right mouse button so that is orientated the same as the data in View A. Hold R + Right mouse button to draw a zoom window in both views to a common feature (e.g. white line, building edge) and select the ‘tie-point pair’ using SHIFT + Left mouse button in both views. A yellow dot with the tie point number is displayed in each view (it doesn’t matter which view is selected first). Click on the Add point pair button OR15057button1+.jpg to accept the tie point pair.
OR15057screenshot4 7.2.6.1.jpg
  • Repeat the tie-point pair selection procedure until at least 4 pairs have been accepted then click on the Register button. The scan data in View B will be reoriented to best match the tie-point pairs. The scan data in View B will be added to the reference view (View A). Check that the standard deviation is acceptable (accurate to <1 m).
OR15057screenshot5 7.2.6.1.jpg
  • Close the Coarse Registration Box. Repeat the coarse registration for each scan position in turn to roughly align all of the scan positions.

Manually translating and rotating

  • In the Project Manager, right-click the VIEWS folder and select ‘New object view...’. Enter a suitable name (e.g. Registration) and press OK. Drag the scan that you will hold fixed (is this example ScanPos001) into the view. In the ‘Select view type...’ dialogue box, choose ‘Other — Single color’ and choose a colour in the drop-down box, then press OK.
OR15057screenshot1 7.2.6.2.jpg
  • Repeat for the next scan (in this example ScanPos002), choosing a different colour. You will now have two scans in the Object View, coloured differently. Press the ‘Bird’s Eye View’ button OR15057birdseyeview.jpg to view both scans from above.
  • Switch to the Object inspector, and under POSITIONS right-click ScanPos002 (not the scan itself) and choose ‘Modify position and orientation’.
OR15057screenshot2 7.2.6.2.jpg
  • The Modify Orientation and Position dialogue box appears. Under ‘Action’, select ‘Rotate’. Hover over the blue ‘Z’ arrow and drag left or right to rotate the scan position around until it lines up with the next scan. (Hint: identify the position of the next scan position in the first scan, and rotate the scan until the scan position)
OR15057screenshot3 7.2.6.2.jpg
  • Then, change the action to ‘Translate’, and drag the red ‘X’ and green ‘Y’ arrows to move the scan around laterally until it matches up with the fixed scan.
  • You may need to switch back to ‘Rotate’ to improve the orientation of the scan once it is closer. Be sure that you ONLY ROTATE on the blue ‘Z’ axis, and you ONLY TRANSLATE on red ‘X’ and green ‘Y’. If you make a mistake or get lost, you can set the scan back to the beginning by double-clicking the ‘SOP’ for the incorrect scan position, and right-clicking the table and selecting “Set default”.
OR15057screenshot4 7.2.6.2.jpg
  • Repeat for all remaining scans, until you have all the scans aligned approximately.

You are now ready to improve the alignment using Multi Station Adjustment (MSA). The MSA algorithm uses filtered versions of each scan to perform the surface matching registration process. A plane filter is used to identify and triangulate planar surfaces in each scan. The MSA process then identifies common planar surfaces from each scan and shifts the scan data until the best match is achieved.

  • From the Registration menu select the Multi Station Adjustment > Prepare data option.
OR15057screenshot5 7.2.6.2.jpg
  • In the Create new polydata dialogue box check all the scans to be plane filtered on the Data tab.
OR15057screenshot6 7.2.6.2.jpg
  • Switch to the Settings tab and check the Plane Surface filter (uncheck any other filter options).
OR15057screenshot7 7.2.6.2.jpg
  • The following settings are recommended for the Plane Surface Filter:
Parameter Value
Max plane error [m] 0.05
Max edge length [m] 1.0
Min range [m] 1.0
Reference range [m] 5.0
Split angle [deg] 20
  • For rural scan scenes with limited planar surfaces it may be necessary to increase the ‘Max plane error’ setting to 0.1–0.2 m in order to identify sufficient planar surfaces.
  • Click OK to create the plane filtered data. The filtered data appears in the POLYDATA branch of each Scan position in the Project manager window. View each filtered dataset (polydata) to check that it is representative of the actual scan.
OR15057screenshot8 7.2.6.2.jpg

Multi station adjustment From the Registration menu select the Multi Station Adjustment > Start adjustment(or press the F4 short cut button).

OR15057screenshot1 7.2.6.3.jpg
  • The Multi Station Adjustment (MSA) dialogue pox opens. The dialogue box displays a list of the scan positions to be registered. Check all the scan positions you wish to adjust.
  • Select the ‘fixed’ scan position, right click and select the Lock position and orientation option.
  • The check boxes next to the X,Y,Z and R,P,Y values are unchecked meaning they will not be changed during the calculation. The Scale check boxes should be unchecked for all scans since it should not be necessary to scale the data.
  • The registration process is iterative, usually requiring 2 or 3 iterations depending on the accuracy of the coarse alignment and the number of scans. The following PARAMETERS are recommended:
Parameter First iteration Subsequent iterations
Mode all nearest points all nearest points
Search radius [m] 1.0 0.5
Max tilt angle [deg] 5.0 5.0
Min. change of error 1 [m] 0.10 0.10
Min. change of error 2 [m] 0.05 0.005
Outlier threshold [1] 2.0 2 x Error[StdDev] *
Calculation mode least square fitting least square fitting
Update display seldom seldom

* The outlier threshold should be set to 2 x Error[StdDev] of each previous iteration

  • The ADJUSTMENT check boxes should not be checked. These are for debug purposes only.
  • Click Calculate to run the first iteration.
  • The first iteration may take several minutes depending on the accuracy of the coarse registration and the number of scan positions. When the calculation is complete the Multi Station Adjustment dialogue box will switch to the Results tab and 3D plane orientation sphere and residues histogram are displayed. The plane orientation sphere shows the spread of orientation of the various planar surfaces used in the calculation. A good spread of orientations is desirable. The histogram shows the spread of errors for matching surfaces. A normal distribution centred around 0.0 m is desirable.
  • The STATISTICS section displays the number of polydata observations used in the calculation. This should be several thousand. The Error [StdDev] value represents the average surface match. A value of less than 0.02 m is desirable. Set the Search radius to 0.5 m and the Outlier threshold to 2 x Error [StdDev] value and repeat the calculation.
  • Once an acceptable Error value is achieved the registration statistics can be saved in the form of a CSV file by clicking the Save calculation statistics to file (*.csv)… button. Close the Multi Station Adjustment dialogue box and visually check the combined point cloud data.

Save the data.

  • In the survey folder (*.RiSCAN) create a new folder named 'DATA'. Right-click each scan. Choose Export, give it a relevant (*.txt) File name, and Save as ASCII (*.*). Click Save. Make sure GLCS appears in the drop-down, Prune invalid points is ticked, xyz RGB (colour) or xyz Amplitude (intensity) are ticked, and the comma button is highlighted. <OK>
  • Alternatively Save as LAS1.2 (*.las). This will automatically save the RGB and Intensity data.
OR15057screenshot2 7.2.6.3.jpg
  • Once All scans have been exported. Save and Close the Project.