Lab 5: Accuracy Assessment of Image Classification

Introduction

  The goal of this lab is to create and interpret an error matrix using Erdas Imagine for the second classified image from Lab 3 using unsupervised classification and for the classified image produced using supervised classification in Lab 4. There are two main steps in creating the matrix: (1) collecting ground reference points and enter in truth values, and (2) creating the matrix from the accuracy points table.

Methods

Step 1: Collect Ground Reference Points and Enter in Truth Values
   First, the unsupervised image will be used to create a matrix, so the unsupervised image was added to a viewer. Then, the reference image was added to a different viewer. The reference image is a higher resolution RGB image of Eau Claire and Chippewa counties. After this, the Accuracy Assessment window was opened for the classified image by navigating to Raster → Supervised → Accuracy Assessment. Then, the classified image was opened in the window. The Select viewer to display random points button was then used to set the reference image. To create the random points in this image, under the edit tab, the Create/Add Random Points... button was clicked to open up the Add Random Points window. In this window, the number of points was set to 125, the sampling scheme was set to stratified random, the minimum number of points per class was set to 15, and the 5 LULC classes were selected to be used only. When these points were created, for some reason, Erdas would only allow for 250 points to be created, so instead of the entered in value of 125 points, 250 points were used in the accuracy assessment. To attempt to eliminate bias in the accuracy assessment, at this moment, the viewer which contained the classified image was deleted as it was no longer needed, and would only be used to artificially increase the accuracy. It is important to collect the reference values without knowing what the classified image classified the points as
 
  To enter in the truth values, the assessment points were added to display 10 at a time to allow the author of this article to more easily find the location of the points. If all the points were added to the display at the same time, it would have taken a long time to find each assessment point because there were 250 of them. LULC values were entered as their classified raster value. This is 1 for water, 2 for forest, 3 for agriculture, 4 for urban/built up, and 5 for bare soil. To verify when a value for a point was entered into the table, the color of the point was changed to turquoise. If a value hadn't been entered in for the truth value yet, the color was set to white. One more important thing to mention here, is that if an assessment point was right on the border between two LULC classes, it was deleted. In total, 12 points were deleted. It is also important to note that to make sure the output matrix has at least 90% confidence, at least 250 assessment points should be assessed. However, only 250 points were used in this lab to save on time. Figure 1 below is a screenshot of the accuracy points and the corresponding table after the truth values had been entered in. Notice how all of the assessment points are turquoise and that the reference values are populated in the appropriate cells. After this, the table was saved.

Fig 1: Making Sure All of the Accuracy Assessment Points Have a Truth Value

Step 2: Creating the Error Matrix
  The error matrix is created by navigating within the Accuracy Assessment window to Report → Accuracy Report.... This produces a report file which can be seen below in Figure 2. This report contains all of the information regarding the overall accuracy, Kappa statistics, producer's accuracy, and the user's accuracy for the classified image. To make an easier to read report, the important information from this report was transposed over into Excel to create an error matrix.

Fig 2: Error Report for Unsupervised Assessment Points

After creating this error matrix in Excel for the unsupervised classified image, the same process was used to created an error matrix in Excel for the supervised classified image.

Results

  Figure 3 shows the error matrix created for the classified image created using unsupervised classification as performed in Lab 3. 

Fig 3: Error Matrix for Classified Image using Unsupervised Classification

  The overall accuarcy of the LULC classification using the unsupervised method is 69%. This is not very good. Generally one would like to classify and image with at least 85% overall accuracy. Looking at the producer and user accuracy, it appears that the bare soil LULC class performed the worst among the LULC classes. This is because it was the most difficult to identify the bare soil when performing the supervised classification as the only part of the image delineated as bare soil were areas that had bare soil, such as around the airport runway. Fallow agriculture fields were classified as agriculture because that is their general land use. It appears that the water LULC class classified the best. It should be noted that the user accuracy is significantly lower for both the urban/built up and bare soil LULC classes than the producer's accuracy is for the classes. The overal Kappa statistic is 0.5706 which means that there was moderate agreement between the accuracy.

  Figure 4 shows the error matrix created for the classified image created using supervised classification as performed in Lab 4.

Fig 4: Error Matrix for Classified Image using Supervised Classification

  The overall accuracy of the LULC classification using the supervised method is 60%. This is not very good either and would be unacceptable if this map were to be used for anything other than for learning how to create a LULC image. Among the LULC classes, water classified the best, followed by forest. The producer's and user's accuracy has the greatest difference between the agriculture, and urban/built up LULC. Urban/built up LULC most likely didn't classify well because the spectral profiles for the training samples (seen in Figure 4 of Lab 4) were not uniform at all. The overall Kappa statistic for this classification is .4582 which means that there was moderate agreement between the accuracy.
  For some reason, the image classified using unsupervised classification had a higher accuracy than the image did which was classified using supervised classification. This is most likely because the image alarm was only used for the water training samples and not for the other LULC samples. Using the image alarm more would have helped to identify the training samples which needed to be refined. Also, the training samples weren't refined all that much in Lab 4. If these training samples had been more homogeneous and of higher quality, it is likely that the supervised classification would have produced  a more accurate classified image that the unsupervised method did.

Sources

United States Geological Survey. (2017). Earth Resources Observation and Science Center
Wilson, C (2017) Lab 5 Accuracy Assessment of Image Classification retrieved from
    https://drive.google.com/open?id=1iSUmx73tjWWJGctIyu5oeHLk90w8ZKTc