Accuracy Assessment Accuracy Assessment Accuracy Assessment Accuracy Assessment

(1)

Chapter 14 Chapter 14

Accuracy Assessment Accuracy Assessment Accuracy Assessment Accuracy Assessment

Introduction to Remote Sensing, Introduction to Remote Sensing,

James B. Campbell

James B. Campbell pp

(2)

O tline O tline Outline Outline

Definisi Definisi

Accuracy & PrecissionAccuracy & Precission

SignificanceSignificance

Source of Classification Error Source of Classification Error

Error Characteristics Error Characteristics

Measurement of Map Accuracy Measurement of Map Accuracy

Error MatrixError Matrix

Omission & CommissionOmission & Commission

User & Producer AccuracyUser & Producer Accuracy

Interpretation of the Error matrix Interpretation of the Error matrix

Percentage CorrectPercentage Correct

Quantitative Assessment of Error matrixQuantitative Assessment of Error matrix

(3)

Definition Definition Definition Definition

Accuracy : correctness, mengukur “kecocokan” antara Accuracy : correctness, mengukur “kecocokan” antara suatu image yg tidak diketahui kualitasnya dengan

suatu image yg tidak diketahui kualitasnya dengan sebuah standar image

sebuah standar image

Precission : detail, “The distinction is important because Precission : detail, “The distinction is important because one may be able to increase accuracy by decreasing

one may be able to increase accuracy by decreasing precission”,

precission”,

Meningkatkan detail = menambah ragam kategori . Misal Meningkatkan detail = menambah ragam kategori . Misal

: forest = caniferous, pine, shortleaf pine atau mature : forest = caniferous, pine, shortleaf pine atau mature shortleaf pine

shortleaf pineÎ Î akan menambah peluang klasifikasi akan menambah peluang klasifikasi error

error error error

Statistical context : high accuracy = low bias, (estimated Statistical context : high accuracy = low bias, (estimated value is consistenrly close to an accepted reference

value is consistenrly close to an accepted reference value)

value)

(4)

Definition

(5)

Definition Definition Definition Definition

Significance Significance

Accuracy has many practical implication : Accuracy has many practical implication : Accuracy has many practical implication : Accuracy has many practical implication :

effect legal standing, operational usefulness, effect legal standing, operational usefulness, validity for scientific research.

validity for scientific research.

(6)

So ce of Classification E o So ce of Classification E o Source of Classification Error Source of Classification Error

Manual Interpretation : Misidentification, Manual Interpretation : Misidentification, Excessive generalization, Error registration, Excessive generalization, Error registration, Variation in detail of interpretation etc.

Variation in detail of interpretation etc.

Character of landscape : parcel size, variation in Character of landscape : parcel size, variation in

l i l id i i b f

parcel size, parcel identities number of parcel size, parcel identities number of

categories, arrangement of categories, number categories, arrangement of categories, number of parcel per category shapes of parcel

of parcel per category shapes of parcel of parcel per category, shapes of parcel, of parcel per category, shapes of parcel, radiometric and spectral contrast with radiometric and spectral contrast with surrounding parcel

surrounding parcel

(7)

So ce of Classification E o So ce of Classification E o Source of Classification Error Source of Classification Error

Three error types dominate:

Data Acquisition Errors: These include sensor performance, stability of Data Acquisition Errors: These include sensor performance, stability of the platform, and conditions of viewing. We can reduce them or

the platform, and conditions of viewing. We can reduce them or compensate for them by making systematic corrections (e g by compensate for them by making systematic corrections (e g by compensate for them by making systematic corrections (e.g., by compensate for them by making systematic corrections (e.g., by calibrating detector response with on

calibrating detector response with on--board light sources generating board light sources generating known radiances). We can make corrections, often modified by

known radiances). We can make corrections, often modified by

ancillary data such as known atmospheric conditions, during the initial ancillary data such as known atmospheric conditions, during the initial processing of the raw data.

processing of the raw data.

p g

Data Processing Errors: An example is misregistration of equivalent Data Processing Errors: An example is misregistration of equivalent pixels in the different bands of the Landsat Thematic Mapper. The goal pixels in the different bands of the Landsat Thematic Mapper. The goal in geometric correction is to hold the mismatch to a displacement of no in geometric correction is to hold the mismatch to a displacement of no more than one pixel. Under ideal conditions, and with as many as 25 more than one pixel. Under ideal conditions, and with as many as 25

d l (GC ) d d l h

ground control points (GCP) spread around a scene, we can realize this ground control points (GCP) spread around a scene, we can realize this goal. Misregistrations of several pixels significantly compromise

goal. Misregistrations of several pixels significantly compromise accuracy.

accuracy.

SceneScene--dependent Errors: As alluded to in the previous page, one such dependent Errors: As alluded to in the previous page, one such error relates to how we define and establish the class which in turn is error relates to how we define and establish the class which in turn is error relates to how we define and establish the class, which, in turn, is error relates to how we define and establish the class, which, in turn, is sensitive to the resolution of the observing system and the reference sensitive to the resolution of the observing system and the reference map or photo. Mixed pixels fall into this category.

map or photo. Mixed pixels fall into this category.

(8)

So ce of Classification E o

Source of Classification Error

(9)

E o Cha acte istics E o Cha acte istics Error Characteristics Error Characteristics

Classification error : assignment pixel to one category Classification error : assignment pixel to one category that different from true category ( as determined

that different from true category ( as determined ground observation/ground

ground observation/ground--truth ). truth ).

Error Characteristic : Error Characteristic :

Error are not distributed over the image at random, display a Error are not distributed over the image at random, display a gg ,, p yp y degree of systematic, ordered occurrence in space.

degree of systematic, ordered occurrence in space.

Often erroneously assigned pixels are not spatially isolated but Often erroneously assigned pixels are not spatially isolated but occur grouped in areas of varied size and shape (Campbell 1981) occur grouped in areas of varied size and shape (Campbell 1981)

Errors may have specific spatial relationships to the parcels toErrors may have specific spatial relationships to the parcels to

Errors may have specific spatial relationships to the parcels to Errors may have specific spatial relationships to the parcels to which they pertain, for example, they may tend to occur at which they pertain, for example, they may tend to occur at edges or in the interiors of the parcels

edges or in the interiors of the parcels

(10)

E o Cha acte istics E o Cha acte istics Error Characteristics Error Characteristics

Tiga macam eror patern dari Landsat, Cogalton (1984). Dark area = error clasification, white area = correct.

(11)

Meas ement of Map Acc ac Meas ement of Map Acc ac Measurement of Map Accuracy Measurement of Map Accuracy

Compare the “true map”/ reference map, Compare the “true map”/ reference map, (asumsi lebih akurat) with image to be

(asumsi lebih akurat) with image to be

( ) g

evaluated.

Jika pembandingan tanpa memperhatikan Jika pembandingan tanpa memperhatikan

Jika pembandingan tanpa memperhatikan Jika pembandingan tanpa memperhatikan posisi pixel, klasifikasi total bisa dianggap posisi pixel, klasifikasi total bisa dianggap sama meskipun sebenarnya posisi dengan sama meskipun sebenarnya posisi dengan sama meskipun sebenarnya posisi dengan sama meskipun sebenarnya posisi dengan image reference tidak sesuai

image reference tidak sesuaiÎ Î site site-- spesific accuracy

spesific accuracy

(12)

Meas ement of Map Acc ac Meas ement of Map Acc ac Measurement of Map Accuracy Measurement of Map Accuracy

Site & Non Site Specific Error Site & Non Site Specific Error

(13)

Meas ement of Map Acc ac Meas ement of Map Acc ac Measurement of Map Accuracy Measurement of Map Accuracy

Error Matrix : matrik perbandingan image Error Matrix : matrik perbandingan image reference dengan image yang akan

reference dengan image yang akan g g g y g g y g

dianalisa berdasarkan kelompok klasifikasi dianalisa berdasarkan kelompok klasifikasi pixel

pixel--pixel yang sama dalam image pixel yang sama dalam image--image image

pp p p y g y g g g gg

tersebut.

tersebut.Î Î dari Error Matrik dapat dari Error Matrik dapat dihitung % correct,

dihitung % correct, g g , ,

% correct = sum agreement pixel between reff & image(jumlah

% correct = sum agreement pixel between reff & image(jumlah diagoal pada error matrix)/total pixel

diagoal pada error matrix)/total pixel

(14)

Meas ement of Map Acc ac Meas ement of Map Acc ac Measurement of Map Accuracy Measurement of Map Accuracy

Error Matrix Error Matrix

(15)

Meas ement of Map Acc ac Meas ement of Map Acc ac Measurement of Map Accuracy Measurement of Map Accuracy

Compiling Error matrix Compiling Error matrix

Image direpresentasikan dengan pixel2 Image direpresentasikan dengan pixel2 Image direpresentasikan dengan pixel2 Image direpresentasikan dengan pixel2

Hitung jumlah pixel untuk tiap klasifikasi Hitung jumlah pixel untuk tiap klasifikasi

Yg perlu diperhatikan : Klasifikasi reference Yg perlu diperhatikan : Klasifikasi reference

Yg perlu diperhatikan : Klasifikasi reference Yg perlu diperhatikan : Klasifikasi reference dengan image yg akan diklasifikasi harus dengan image yg akan diklasifikasi harus compatible

compatibleÎ pp Î turunan klasifikasi harus masih turunan klasifikasi harus masih sesuai dengan kategori pada reference

sesuai dengan kategori pada reference

(16)

(17)

Meas ement of Map Acc ac

Measurement of Map Accuracy

(18)

Meas ement of Map Acc ac Meas ement of Map Acc ac Measurement of Map Accuracy Measurement of Map Accuracy

Omission & Commission Error Omission & Commission Error

Omission : jumlah pixel pada reference image yang tidak sesuai Omission : jumlah pixel pada reference image yang tidak sesuai dengan kategori kalsifkasi pada image yg dievaluasi

dengan kategori kalsifkasi pada image yg dievaluasigg gg pp g ygg yg

Commission : jumlah pixel pada image yang dievaluasi yang Commission : jumlah pixel pada image yang dievaluasi yang tidak sesuai dengan keadaan sebanarnya/klasifkasi pada tidak sesuai dengan keadaan sebanarnya/klasifkasi pada reference

reference

CA (Customer Accuracy) & PA (Produsen Accuracy) CA (Customer Accuracy) & PA (Produsen Accuracy)

CA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan CA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan kondisi reference dibandingkan jumlah total pixel pada image yg kondisi reference dibandingkan jumlah total pixel pada image yg

d l k kl f k b

dievaluasi untuk klasifikasi tsb.

PA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan PA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan kondisi reference dibandingkan jumlah total pixel pada image kondisi reference dibandingkan jumlah total pixel pada image reference

reference reference.

reference.

(19)

Meas ement of Map Acc ac

Measurement of Map Accuracy

(20)

http://rst.gsfc.nasa.gov/Sect13/Sect13_3.html

(21)

Interpretation of the Error Interpretation of the Error pp

matrix matrix

Percentage Correct (PC) Percentage Correct (PC)

Ukuran yg sering dipakai Ukuran yg sering dipakai

Beberapa rekomendasi : Beberapa rekomendasi :

PC = 85 % dibutuhkan untuk landPC = 85 % dibutuhkan untuk land--use data resource use data resource management (Anderson et al, 1976)

management (Anderson et al, 1976) management (Anderson et al, 1976) management (Anderson et al, 1976)

FitzpatrickFitzpatrick--Lins(1978) akurasi dari USGS landLins(1978) akurasi dari USGS land--cover map cover map

untuk central Atlantic coastal : 85 % (untuk skala 1:24.000), untuk central Atlantic coastal : 85 % (untuk skala 1:24.000), 77% (1:100.000), 73% (1:250.000)

77% (1:100.000), 73% (1:250.000) 77% (1:100.000), 73% (1:250.000) 77% (1:100.000), 73% (1:250.000)

Untuk automasiUntuk automasi--interpretasi dari Landuse menggunakan interpretasi dari Landuse menggunakan hanya data MSS PC yang didapat = 38%, dan untuk MSS + hanya data MSS PC yang didapat = 38%, dan untuk MSS + ancillary data PC = 78 % (Tom et al. 1978)

ancillary data PC = 78 % (Tom et al. 1978) ancillary data PC 78 % (Tom et al. 1978) ancillary data PC 78 % (Tom et al. 1978)

(22)

http://rst.gsfc.nasa.gov/Sect13/Sect13_3.html

(23)

I t t ti f th E t i

Interpretation of the Error matrix Interpretation of the Error matrix

Quantitative Assessment of the Error Matrix Quantitative Assessment of the Error Matrix kappa (k) = measured of difference between

kappa (k) = measured of difference between observed observed tt b t b t t t d d th th t th t t th t agreement

agreement between two map and between two map and the agreement that the agreement that might be attained solely by chance matching two map might be attained solely by chance matching two map..

k = (observed

k = (observed ––expected)/(1 expected)/(1-- expected) expected) Observed = percentage correct

Observed = percentage correct

Expected = product row & column,

Expected = product row & column,Î Î change agreement change agreement two categories when two images superimposed

two categories when two images superimposed (Fig (Fig 14 8)

14 8)

14.8)

(24)

I t t ti f th E t i

Interpretation of the Error matrix

(25)

(26)

I t t ti f th E t i

Interpretation of the Error matrix Interpretation of the Error matrix

k = 0 83 k = 0 83 Î Î accuracy = 83% better than accuracy = 83% better than

k 0.83 k 0.83 Î Î accuracy 83% better than accuracy 83% better than expected from chance assignment of pixel expected from chance assignment of pixel to cattegories

to cattegories to cattegories.

to cattegories.

k = +1, k = +1,Î Î accuracy = 100%, perfect accuracy = 100%, perfect classification table 14 6

classification table 14 6

classification, table 14.6

(27)

(28)

Pen t p Pen t p Penutup Penutup

Accuracy dibutuhkan sebagai ukuran informasi yang didapatkan Accuracy dibutuhkan sebagai ukuran informasi yang didapatkan mendekati nilai standar/referensi tertentu/nilai sebenarnya

mendekati nilai standar/referensi tertentu/nilai sebenarnya

Untuk aplikasi tertentu direkomendasikan menggunakan suatu nilai Untuk aplikasi tertentu direkomendasikan menggunakan suatu nilai

Untuk aplikasi tertentu direkomendasikan menggunakan suatu nilai Untuk aplikasi tertentu direkomendasikan menggunakan suatu nilai accuracy tertentu. Selain itu accracy juga berdampak pada nilai accuracy tertentu. Selain itu accracy juga berdampak pada nilai legal dari data dan informasi yang dihasilkan.

legal dari data dan informasi yang dihasilkan.

Accuracy didapatkan dengan membandingkan dengan suatu image Accuracy didapatkan dengan membandingkan dengan suatu image referensi tertentu, yg dianggap benar, lebih akurat dst

referensi tertentu, yg dianggap benar, lebih akurat dst

Untuk mengukur accuracy digunakan alat bantu error matrix Untuk mengukur accuracy digunakan alat bantu error matrix

Untuk mengukur accuracy digunakan alat bantu error matrix, Untuk mengukur accuracy digunakan alat bantu error matrix,

dengan menghitung percentage correct, omission&comission error, dengan menghitung percentage correct, omission&comission error, PA & CA dan kappa, semuanya untuk melihat kerelatifan kebenaran PA & CA dan kappa, semuanya untuk melihat kerelatifan kebenaran klasifkasi yang telah dilakukan.

klasifkasi yang telah dilakukan.