Finding corners

(1)

4A-L2 Finding corners

CS4495/6495

Introduction to Computer Vision

(2)

Feature points

(3)

Characteristics of good features

Repeatability/Precision

•

The same feature can be found in several images

despite geometric and photometric transformations

(4)

Saliency/Matchability

•

Each feature has a distinctive description

(5)

Characteristics of good features

Compactness and efficiency

•

Many fewer features than image pixels

(6)

Locality

•

A feature occupies a relatively small area of the

image; robust to clutter and occlusion

(7)

Corner Detection: Basic Idea

“edge”:

no change

along the edge direction

“corner”:

significant change in all directions with small shift

“flat” region:

no change in all directions

Source: A. Efros

(8)

Finding Corners

•

Key property: in the region around a corner, image

gradient has two or more dominant directions

(9)

Finding Corners

C. Harris and M. Stephens. "A Combined Corner and Edge

Detector,” Proceedings of the 4th Alvey Vision Conference: 1988

(10)

Corner Detection: Mathematics

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

Intensity Shifted

intensity Window

function

Window function w(x,y) = or

Gaussian 1 in window,

0 outside

Source: R. Szeliski

Change in appearance for the shift [u,v]:

(11)

Corner Detection: Mathematics

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

I(x, y)

E(u, v)

E(0,0)

E(3,2)

(12)

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

We want to find out how this function behaves for small shifts (u,v near 0,0)

(13)

Second-order Taylor expansion of E(u,v) about

(0,0) (local quadratic approximation for small u,v):

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

(14)

Corner Detection: Mathematics

2 2

2

( 0 ) 1 ( 0 )

( ) ( 0 ) · ·

2

d F d F

F x F x x

d x d x

     

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

(15)

( 0 , 0 ) 1 ( 0 , 0 ) ( 0 , 0 )

( , ) ( 0 , 0 ) [ ] [ ]

( 0 , 0 ) 2 ( 0 , 0 ) ( 0 , 0 )

u u u u v

v u v v v

E E E u

E u v E u v u v

E E E v

     

        

 

   

2 2

2

( 0 ) 1 ( 0 )

( ) ( 0 ) · ·

2

d F d F

F x F x x

d x d x

     

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

(16)

( 0 , 0 ) 1 ( 0 , 0 ) ( 0 , 0 )

( , ) ( 0 , 0 ) [ ] [ ]

( 0 , 0 ) 2 ( 0 , 0 ) ( 0 , 0 )

u u u u v

v u v v v

E E E u

E u v E u v u v

E E E v

     

  

      

   

Second-order Taylor expansion of E(u,v) about (0,0):

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

(17)

( 0 , 0 ) 1 ( 0 , 0 ) ( 0 , 0 )

( , ) ( 0 , 0 ) [ ] [ ]

( 0 , 0 ) 2 ( 0 , 0 ) ( 0 , 0 )

u u u u v

v u v v v

E E E u

E u v E u v u v

E E E v

     

  

      

   

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

Need these derivatives…

(18)

 

,

( , ) 2 ( , ) ( , ) ( , ) ( , )

u u v x y w x y I x u y v

E ^  ^ ^ ^ I x y I _x x ^ u y ^ v

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

(19)

 

,

( , ) 2 ( , ) ( , ) ( , )

2 ( , ) ( , ) ( , ) ( , )

x y

u u u v w x y x u y v x u y v

w x y I x u y v I x y x u v

E

y

I I

I

     

      

x x

x x

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

(20)

 ²

,

( , ) ( , ) ( , ) ( , )

x y

E u v   w x y I x  u y  v  I x y

Second-order Taylor expansion of E(u,v) about (0,0):

 

,

( , ) 2 ( , ) ( , ) ( , )

2 ( , ) ( , ) ( , ) ( , )

x y

u v y x

x y

E u v w x y x u y v x u y v

w x y I x u y v I x y x u y v

I I

I

     

      

(21)

 

,

( , ) 2 ( , ) ( , ) ( , ) ( , )

( , ) 2 ( , ) ( , ) ( , )

2 ( , ) ( , ) ( , ) ( , )

( , ) 2 ( , ) ( , ) ( , )

2 ( , ) ( , ) ( ,

u x

x y

u u x x

x y

x x x y

u v y x

x y

E u v w x y I x u y v I x y I x u y v E u v w x y I x u y v I x u y v

w x y I x u y v I x y I x u y v E u v w x y I x u y v I x u y v

w x y I x u y v I x y

      

     

      

     

      ⁾ I x y ⁽ x  u y^,  v ⁾

( 0 , 0 ) 1 ( 0 , 0 ) ( 0 , 0 )

( , ) ( 0 , 0 ) [ ] [ ]

( 0 , 0 ) 2 ( 0 , 0 ) ( 0 , 0 )

u u u u v

v u v v v

E E E u

E u v E u v u v

E E E v

     

  

      

   

(22)

( 0 , 0 ) 1 ( 0 , 0 ) ( 0 , 0 )

( , ) ( 0 , 0 ) [ ] [ ]

( 0 , 0 ) 2 ( 0 , 0 ) ( 0 , 0 )

u u u u v

v u v v v

E E E u

E u v E u v u v

E E E v

     

  

      

   

 

,

( 0 , 0 ) 2 ( , ) ( , ) ( , ) ( , )

( 0 , 0 ) 2 ( , ) ( , ) ( , )

2 ( , ) ( , ) ( , ) ( , )

( 0 , 0 ) 2 ( , ) ( , ) ( , )

2 ( , ) ( , ) ( , ) ( , )

u x

x y

u u x x

x y

x x x y

u v y x

x y

x y x y

E w x y I x y I x y I x y

E w x y I x y I x y

w x y I x y I x y I x y

  

 

  

 

  

Evaluate E and its derivatives at (0,0):

= 0

(23)

( 0 , 0 ) 0 ( 0 , 0 ) 0 ( 0 , 0 ) 0

u

v

E E E



( 0 , 0 ) ( 0 , 0 )

(

1 ( 0 , 0 ) ( 0 , 0 )

( , ) [ ] [ ]

( 0 , 0 ) ( 0 , 0 )

0 2

0 , )

u u u v

u v v v

u

v

E

E E E u

E u v u v u v

v

E E E

   

  

 

 

 

    

 

,

( 0 , 0 ) 2 ( , ) ( , ) ( , )

u u x x

x y

v v y y

x y

u v x y

x y

 

(24)

2

, ,

2

, ,

( , ) ( , ) ( , ) ( , ) ( , )

( , ) [ ]

( , ) ( , ) ( , ) ( , ) ( , )

x x y

x y x y

x y y

x y x y

w x y I x y w x y I x y I x y E u v u v u

w x y I x y I x y w x y I x y v

 

   

       

( 0 , 0 ) 0 ( 0 , 0 ) 0 ( 0 , 0 ) 0

u

v

E E E



,

( 0 , 0 ) 2 ( , ) ( , ) ( , )

u u x x

x y

v v y y

x y

u v x y

x y

 

(25)

The quadratic approximation simplifies to

2

2 ,

( , ) ^x ^x ^y

x y x y y

I I I

M w x y

I I I

 

  

 

 



where M is a second moment matrix computed from image derivatives:

( , ) [ ] u

E u v u v M

v

  

   

(26)

The second moment matrix M:

2

2 ,

( , ) ^x ^x ^y

x y x y y

I I I

M w x y

I I I

 

  

 

 



Can be written (without the weight):

Each product is a rank 1 2x2

( )

x T

x x x y

x y

x y y y y

I I I I I

M I I I I

I I I I I

     

 

            

     

 

   

 

(27)

The surface E(u,v) is locally approximated by a quadratic form.

Interpreting the second moment matrix

( , ) [ ] u

E u v u v M

v

  

   

2

2 ,

( , ) ^x ^x ^y

x y x y y

I I I

M w x y

I I I

 

   

(28)

Consider a constant “slice” of E(u, v):

Interpreting the second moment matrix

[ ] u c o n s t

u v M

  v

   This is the equation of an ellipse.

2 2 2 2

x 2 x y y

I u I I u v I v k

     

(29)

2

1 2

, 2

( , ) 0

0

x x y

x y x y y

I I I

M w x y

I I I



   

          

First, consider the axis-aligned case where gradients are either horizontal or vertical

If either λ is close to 0, then this is not a corner, so look for locations where both are large.

Interpreting the second moment matrix

(30)

1 1

2

0

M R  0 R





 

    

The axis lengths of the ellipse are determined by the eigenvalues and the orientation is

determined by R

Diagonalization of M:

(31)

direction of the slowest change direction of the

fastest change

(_max)^-1/2

(_min)^-1/2

(32)

Interpreting the eigenvalues



₁



₂

“Corner”

₁ and ₂ are large,

₁~ ₂;

E increases in all directions

₁ and ₂ are small;

E is almost constant in all directions

“Edge”

₁ >> ₂

“Edge”

₂ >> ₁

“Flat”

region

Classification of image points using eigenvalues of M:

(33)

Harris corner response function

“Corner”

R > 0

“Edge”

R < 0

“Edge”

R < 0

“Flat”

region

|R| small

α: constant (0.04 to 0.06)

2 2

1 2 1 2

d e t ( ) t r a c e ( ) ( )

R  M 



M 

 



 





(34)

Harris corner response function

“Corner”

R > 0

“Edge”

R < 0

“Edge”

R < 0

“Flat”

region

|R| small

R depends only on

eigenvalues of M, but

don’t compute them (no sqrt, so really fast even in the ’80s).

2 2

1 2 1 2

d e t ( ) t r a c e ( ) ( )

R  M 



M 

 



 





(35)

Harris corner response function

“Corner”

R > 0

“Edge”

R < 0

“Edge”

R < 0

“Flat”

region

|R| small

2 2

1 2 1 2

d e t ( ) t r a c e ( ) ( )

R  M 



M 

 



 





R is large for a corner

R is negative with large

magnitude for an edge

|R| is small for a flat

region

(36)

Low texture region

Gradients have small magnitude => small ₁, small ₂

( )^T M    I  I

(37)

Edge

Large gradients, all the same => large ₁, small ₂

( )^T M    I  I

(38)

High textured region

Gradients different, large magnitudes => large ₁, large ₂

( )^T M    I  I

(39)

Harris Detector: Algorithm

1. Compute Gaussian derivatives at each pixel

2. Compute second moment matrix M in a Gaussian window around each pixel

3. Compute corner response function R

4. Threshold R

5. Find local maxima of response function (nonmaximum suppression)

C. Harris and M. Stephens. "A Combined Corner and Edge Detector.“

Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.

(40)

Harris Detector: Workflow

(41)

Harris Detector: Workflow

Compute corner response R

(42)

Find points with large corner response: R>threshold

(43)

Take only the points of local maxima of R

(44)

(45)

Other corners:

Shi-Tomasi ’94:

“Cornerness” = min (λ₁, λ₂) Find local maximums cvGoodFeaturesToTrack(...)

Reportedly better for region undergoing affine deformations

(46)

Other corners:

• Brown, M., Szeliski, R., and Winder, S. (2005):

• There are others…

0 1

d e t t r

M M

 

 