OR/15/001 Validation results

Jordan, J J, Grebby, S, Dijkstra, T, Dashwood, C and Cigna, F. 2015. Risk information services for Disaster Risk Management (DRM) in the Caribbean. British Geological Survey Internal Report, OR/15/001.

Quality checking

Service 1: Land use/land cover mapping

Initial quality checks comprised evaluating whether the land use/land cover products satisfied the minimum requirements outlined in the SOW. Regarding spatial coverage, the minimum coverage of 80% of the AOIs was achieved by using the RapidEye imagery and existing land use/land cover maps to infill areas significantly obscured by cloud and shadowing in the Pleiades imagery. As a result, the land use/land cover maps cover 100% of the AOI.

Water features (e.g. lakes, ponds, rivers, streams) and the basic road network could only be delineated on the pan-sharpened Pleiades imagery. Therefore, the completeness of the water features and road vector layers was dependent on cloud and shadow in the Pleiades imagery, as well as the areal coverage of any existing relevant information. Although mapping of such features was hindered by cloud and shadow — resulting in a somewhat fragmented dataset in some areas — the vector layers satisfy the minimum coverage requirements 80%. The satellite imagery used for mapping in this service was required in the last 3 years.

The thematic accuracy of the land use/land cover maps was determined using the conventional remote sensing approach of deriving confusion matrices (Congalton, 1991The confusion matrices for the three AOIs are shown in Tables 8–10. In all cases, the overall accuracies of the maps achieve the target accuracy of 80–90%, with accuracies for St. Lucia, Grenada, and St. Vincent and the Grenadines of 84.9%, 84.8% and 80.8%, respectively. These accuracies were further confirmed through point-location field observations made in St. Lucia and Grenada during validation of the landslide inventories produced in Service 2.

The confusion matrices provide a useful insight into the confusion between classes that affect the land use/land cover maps. From Tables 8–10, it is clear that most of the confusion occurs between different types of vegetation, particularly forests. This can be expected because many of the classes are inherently similar. In St. Lucia, for example, 'Lowland forest' and 'Evergreen forest' both contain Evergreen trees, thus accounting for the observed confusion. Likewise, in Grenada, similarities and overlap between results in some degree of confusion between classes such as 'Semi-deciduous forest', 'Drought Deciduous open woodland', 'Deciduous, coastal Evergreen and mixed forest or shrubland' and 'Evergreen and seasonal Evergreen forest'. Enhanced discrimination between these classes would largely depend on acquiring satellite imagery to capture seasonal variations (i.e. leaf-on and leaf-off conditions). To a lesser degree, confusion between 'Bare ground', 'buildings' and 'Roads and other built-up surfaces' exists in all three maps. Again, this is not entirely unexpected because of colour similarities between these land use/land cover types. For example, bare rock or soil is a similar colour to buildings with reddish roofs, in the imagery, whereas grey buildings appear similar to surfaces made of concrete or asphalt roads. Similarly, some non-metalled or dusty roads also appear similar to bare rock or soil. Enhanced discrimination of these land use/land cover types would require more spectral information than that captured by the 4 multispectral bands of the Pleiades sensor.

Nevertheless, observations both in the field and imagery confirm that this type of confusion has been reduced considerably in the new land use/land cover maps in comparison to the existing maps (Figure 45). However, it is worth noting that a relatively minor degree of confusion does still persist around the coastline, marking the transition from land to the sea. This is most likely the result of the colour similarities between roads, light grey buildings and a sub-pixel mixing of sand/rock and the wake caused by breaking waves on the coast.

Figure 45 Confusion between buildings, roads and bare ground. Left image is Pleiades satellite data, middle is the EO-RISC land cover map, right image is the 2001 land cover map. Includes material ©CNES 2014, Distribution Airbus DS/SPOT Image S.A. France, all rights reserved.

Table 8 Confusion matrix for the St. Lucia land use/land cover map. See Table 4 for full class names.

	Validation data															User’s accuracy (%)
Mapped as	Water	Pasture	Semi‐decid	Lowland	Buildings	Roads	Bare ground	Elfin	Evergreen forest	Golf course	Mangrove	Wetland	Quarry	Woody agric
	1249	0	0	0	19	83	0	0	8	0	0	0	0	0		91.9
Pasture	0	740	96	149	25	6	69	444	81	0	0	0	0	77		79.8
Semi‐decid	0	120	6003	942	0	0	0	0	79	0	138	0	0	527		76.9
Lowland	0	0	112	3886	0	9	2	0	353	0	25	0	0	570		78.4
Buildings	0	110	0	1	845	74	45	0	0	0	0	0	0	0		78.6
Roads	0	0	0	0	180	1501	151	0	0	0	0	0	0	0		81.9
Bare ground	0	64	0	0	96	17	292	0	0	0	0	0	0	0		62.3
Elfin	0	0	0	0	0	0	0	1703	179	0	0	0	0	0		90.5
Evergreen	0	0	0	0	0	0	0	232	2901	0	0	0	0	0		92.6
Golf	0	0	0	0	0	0	0	0	0	1048	0	0	0	0		100.0
Mangrove	0	0	0	0	0	0	0	0	0	0	2712	0	0	0		100.0
Wetland	0	0	0	0	0	0	0	0	0	0	0	590	0	0		100.0
Quarry	0	0	0	0	0	0	0	0	0	0	0	0	619	0		100.0
Woody agric	0	0	0	0	0	0	0	0	0	0	0	0	0	1583		100.0
Producer’s accuracy (%)	100.0	92.7	96.6	78.1	72.5	88.8	52.2	71.6	80.6	100.0	94.3	100.0	100.0	57.4
Overall accuracy (%)	84.9% (K = 0.83)

Table 9 Confusion matrix for the Grenada land use/land cover map. See Table 4 for full class names.

	Validation data																User’s accuracy (%)
Mapped as	Nutmeg	Pasture	Drought	Decid	Semi‐decid	Evergreen	Elfin	Bare ground	Buildings	Roads	Golf course	Wetland	Mangrove	Quarry	Water
Nutmeg	2047	393	13	168	163	262	0	1	28	0	0	0	21	1	0		66.1
Pasture	52	2126	0	224	14	0	0	120	52	3	0	0	2	0	0		82.0
Drought	0	0	622	7	3	0	0	0	0	0	0	0	13	0	0		96.4
Decid	95	365	175	5013	395	4	0	6	25	1	0	0	44	1	0		82.0
Semi‐decid	381	41	4	72	1733	185	0	0	1	0	0	0	4	0	0		71.6
Evergreen	471	69	13	235	212	5283	0	0	0	0	0	0	12	0	0		83.9
Elfin	101	54	0	0	0	73	3910	0	0	0	0	0	0	0	0		94.5
Bare ground	0	1	0	9	0	0	0	1073	9	35	0	0	0	0	0		95.2
Buildings	0	0	0	11	0	1	0	168	1103	56	0	0	0	0	0		83.1
Roads	0	0	0	0	0	0	0	99	133	1123	0	0	0	0	0		82.1
Golf course	0	0	0	0	0	0	0	0	0	0	296	0	0	0	0		100.0
Wetland	0	0	0	0	0	0	0	0	0	0	0	860	0	0	0		100.0
Mangrove	0	0	0	0	0	0	0	0	0	0	0	0	1699	0	0		100.0
Quarry	0	0	0	0	0	0	0	0	0	0	0	0	0	437	0		100.0
Water	0	0	0	0	0	0	0	0	7	3	0	0	9	0	1305		98.6
Producer’s accuracy (%)	65.0	69.7	75.2	87.3	68.8	91.0	100.0	73.1	81.2	92.0	100.0	100.0	94.2	99.5	100.0
Overall accuracy (%)	84.8% (K = 0.83)

Table 10 Confusion matrix for the St. Vincent and the Grenadines land use/land cover map, See Table 4 for full class names.

	Validation data																User’s accuracy (%)
Mapped as	Evergreen	Seasonal	Semi‐decid	Pasture	Bare ground	Roads	Buildings	Drought	Water	Elfin	Woody agric	Montane	Blue Maho	Golf course	Mangrove
Evergreen	2882	1088	1419	5	2	9	15	86	0	283	0	0	0	0	0		49.8
Seasonal	11	867	37	0	0	0	0	3	0	2	0	0	0	0	0		94.2
Semi‐decid	928	273	2764	315	0	0	0	1239	0	25	0	0	0	0	0		49.9
Pasture	29	0	25	7023	866	26	85	482	0	1271	259	0	0	0	0		69.8
Bare ground	0	0	1	5	3450	8	15	4	0	0	0	14	0	0	0		98.7
Roads	0	0	0	2	258	1501	169	0	0	0	0	0	0	0	0		77.8
Buildings	0	0	0	12	52	64	1693	0	0	0	0	0	0	0	0		93.0
Drought	42	82	436	85	99	10	12	10954	0	540	0	0	0	0	0		89.3
Water	0	0	0	21	1	0	0	0	2131	0	0	0	0	0	0		99.0
Elfin	0	0	0	4	0	0	0	0	0	4992	0	234	9	0	0		95.3
Woody agric	0	0	226	0	0	0	0	0	0	0	4351	0	0	0	0		95.1
Montane	0	0	0	0	0	0	0	0	0	40	0	506	0	0	0		92.7
Blue Maho	0	0	0	0	0	0	0	0	0	0	0	0	484	0	0		100.0
Golf course	0	0	0	0	0	0	0	0	0	0	0	0	0	463	0		100.0
Mangrove	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3224		100.0
Producer’s accuracy (%)	74.0	37.5	56.3	94.0	73.0	92.8	85.1	85.8	100.0	69.8	94.4	67.1	98.2	100.0	100.0
Overall accuracy (%)	80.8% (K = 0.78)

Service 2: Hazard mapping to support landslide risk assessment

Landslide inventories

The identification of landslide polygons was quality checked by two operators independently mapping two test areas in St Lucia. The results were compared and this led to fine-tuning of interpretation of indicators and establishment of observational guidance (landuse, bare soil signatures, use of elevation models, minimum landslide size, and use of landslide clusters). The final products were again cross-checked by two engineering geologists. Following the establishment of the multi-temporal inventory the results were evaluated against the 2010 landslide inventory. It was observed that there were some differences between the multi- temporal inventory and the 2010 inventory and these were all carefully evaluated. Generally, these differences were the result of an over-emphasis on ‘bare-earth’ mapping in the 2010 inventory. Additional differences were observed with the more conservative outlines of landslide areas in the multi-temporal inventory as long runouts were not mapped.

The completeness of landslide inventory is a function of the capability of the mode of recording of all the events over a period of time. Clearly, there are several issues therefore that influence completeness. In the case of this project these include, aspects of image resolution, of scale, and of the length of time over which observations are made. Visibility (absence of cloud cover, canopy overhangs, shadow effects, etc.) further influences the completeness of the landslide inventory.

To achieve an insight into the completeness of the landslide inventory of St Lucia, the complete multi-temporal dataset has been analysed. The extent of an individual landslide throughout the multi-temporal dataset is not constant. Deviations from this extent will occur in different years. This can occur through, for example, re-establishment of a vegetation cover leading to increasingly smaller areas remaining ‘active’. But it is also possible that slides extent, either downslope as a consequence of continuing displacement of a landslide mass, or upslope and laterally through retrogressive failure involving increasing large amounts of slope. For this exercise, landslide polygons offer an outline representing the maximum extent of an event throughout the time series. It is therefore unrealistic to construct area frequency diagrams for each year.

The analysis of the full dataset (incorporating the dataset mapped at 1:10k) resulted in a cumulative frequency-area distribution shown in Figure 46, following an approach outlined by Guzzetti 2005 and Malamud et al 2004 (see also Hurst et al. 2014). It is interesting that spill-overs (i.e where the largest number of events are found of a particular size) depend on the nature of the survey. For quick reconnaissance surveys this spill-over lies at a characteristic size of about 104 to 105m² (i.e. landslide dimensions of approximately 100x100m to 300x300m). For more detailed geomorphological surveys greater detail can be captured and this results in a trend with a spill-over at about 103m² (indicative dimensions of about 30x30m) which is similar to what was found in our survey.

Figure 46 a) Cumulative frequency-area distribution for landslide events in St Luci for the period 2010-2014 and b) an example of the different cumulative frequency-area distributions for reconnaissance and for detailed geomorphological surveys (Guzzetti 2005).

Even though from the above it appears that the established inventory provides a degree of confidence that it forms a fair representation of events there are known omissions from the database.

Minimum dimensions in the database are approximately 200 m² and many smaller events are known to have occurred. Very small (<5 m) and obscured (in the shade, on steep slopes, below overhanging vegetation) landslides are difficult to capture but are known to exist in substantial numbers. These small events have a significant impact on lives and livelihoods and recording these through different means will be important to add to the database. It is possible to interrogate the recent Pleiades images (at resolution of 0.5m) at a greater level of detail and it is therefore advantageous to carry out future assessments at scales of 1:5 000 or better. The case study of Chateau Belair (discussed in the Executive summary) highlights the restrictions in using satellite images for interpretation. Subtle changes in topography can be interpreted by skilled operators as signatures of landslides in a complex terrain. At high resolutions (1:10k and better) this can lead to very detailed maps showing geomorphology and engineering geology features relevant to slope instability.

It will remain very difficult to identify very large, relic and inactive landslides. This exercise has shown that very large, and mostly relic, landslides are often difficult to identify and interpret in the context of Caribbean Islands where a substantial vegetation cover masks topographic features.

This will require much detailed additional geological and geomorphological investigations that falls beyond the current scope of the project.

Digital Elevation Models

The accuracy of the 30 DEMs was derived by comparing the elevation values in the DEMs to the most accurate ancillary elevation data available for that AOI using. For St. Lucia, the ancillary elevation data comprised 18 GPS control points. For Grenada, ancillary elevation data comprised a random sample of 500 LiDAR data points selected independently of those used in the vertical calibration. Accordingly, the vertical accuracy of the DEMs was computed as the root-mean-square (RMS) error:

where Z_DEM and Z_ref are the corresponding DEM and ancillary elevations, respectively, and n is the number of elevation values.

Subsequently, the vertical accuracy of the DEM for St. Lucia was found to be 1.4m, whereas the vertical accuracy for the Grenada DEM was computed as 4.9m. The accuracies of these DEMs therefore exceed the target accuracy of 5–10m.

Service 3: Digital Elevation Models of Belize

National Digital Elevation Model

The 20m DEM provides 40% coverage, and the 30m DEM provides 100% coverage of the Belize DEM; thus exceeding the minimum requirement of 80%. The vertical accuracy (RMS) of the 20m DEM is reported to be in the region of 7m by the data providers. With no other ancillary elevation data available, the 20m DEM was used compute the vertical accuracy of the 30m DEM as in Eq. 3 above. As a result, the 30m DEM was found to have a vertical accuracy (RMS) of 9.8m, which is within the target accuracy of 5–10m.

Precise Digital Elevation Model

As discussed above, the precise 1m DEM for a 100km² subset of Belize has not yet been produced due to a lack of suitable tri-stereo Pleiades imagery.

Initial Validation

Preliminary data have only recently been released to the ITC project therefore we do not have validation data from the users yet. A workshop/training exercise with the users is planned for end February to early March 2015 and BGS expects to gain more complete feedback from the users at that workshop/exercise.