finding a chessboard

Finding A Chessboard: An

Introduction To Computer Vision

Motivation & Inspiration

• A spare time project, because I thought it

would be fun

• Project: Chess against the computer,

where board and your pieces are real, it’s

pieces are projected

Requirements

• Interaction should be as natural as possible

– E.g. pieces don’t have to be centered in their squares, or even completely in them

• Should be easy to set up and give demonstrations

– Little calibration as possible

– Work in many lighting conditions

– Although only with this board and pieces

• Camera needs to be at angle to board

• Board made by my father just before he met my mother

– My brother and I learned to play on it

Computer Vision ’70s to ’80s:

Feature Detection

• Initial Idea: Corners are unique, look for them

• Compute the “cornerness” at each pixel by adding

the values of some nearby pixels, and subtracting others, in this pattern:

+ + +

+ + +

+ + +

-- -- -- + + +

• Constant Image (e.g. middle of square): output = 0

• Edge between two regions (e.g. two squares side-by-side):

Corner Detector In Practice

Problems With Corner Detection

• Edge effects

• Strong response for some non-corners

• Easily obscured by pieces, hand

Edge Finding

• Very common in early computer vision

– Early computers didn’t have much power – Very early: enter lines by hand

– A little later: extract line drawing from image • Basic idea: for vertical

edge, subtract pixels on left from pixels on right • Similarly for horizontal

Edge Finding

• Need separate mask for each orientation?

• No! Can compute intensity gradient from horizontal & vertical gradients

– Think of intensity as a (continuous) function of 2D position:

f(x,y)

– Rate of change of intensity in direction (u, v) is – Magnitude changes as cosine of (u, v)

– Magnitude maximum when (u, v) equals (∂f/∂x, ∂f/∂y)

• (∂f/∂x, ∂f/∂y) is called the gradient

– Magnitude is strength of line at this point – Direction is perpendicular to line

Localizing Line

• How do we decide where

lines are & where they

aren’t?

• One idea: threshold the

magnitude

– Problem: what threshold to use? Depends on lighting, etc.

Laplacian

• Better idea: find the peak

– i.e. where the 2nd derivative

crosses zero 2 2 2 2

:

y

f

x

f

Laplacian











• Image shows magnitude

Extracting Whole Lines

• So Far: intensity image  “lineness” image

• Next: “lineness” image  list of lines

• Need to accumulate contributions from across image

– Could be many gaps

• Want to extract position & orientation of lines

Hough Transform

• Parameterize lines by

angle and distance to center of image

• Discretize these and create a 2D grid

covering the entire range

• Each unsuppressed pixel is part of a line

– Perpendicular to the gradient

– Add it’s strength to the

θ

Hough Transform

Angle

D

is

ta

nc

e

To

C

en

te

Coordinate Systems

• In 2D (

u

,

v

) (i.e. on the screen):

– Origin at center of screen

– u horizontal, increasing to the right – _{v vertical, increasing down}

– Maximum u and v determined by field of view.

• In 3D (

x

,

y

,

z

) (camera’s frame):

– Origin at eye

2D



3D

• Perspective projection

– (u, v): image coordinates (2D) – (x, y, z): world coordinates (3D) – Similar Triangles

• Free to assume virtual screen at d = 1

– Relation: u = x/z, v = y/z

2D 3D

p = (u, v) p = (x, y, z)

Lines in 3D map to lines on screen

• Proof: the 3D line, plus the origin (eye),

form a plane.

• All light rays from the 3D line to the eye

are in this plane.

2D 3D

p = (u, v) p = (x, y, z)

u = x/z, v=y/z

From 2D Lines To 3D Lines

• If a group of lines are parallel in 3D, what’s

the corresponding 2D constraint?

• Equation of line in 2D: Au + Bv + C = 0

• Substituting in our formula for u & v:

– Ax/z + By/z + C = 0 – Ax + By + Cz = 0

• A 3D plane through the origin containing the 3D line

2D 3D

p = (u, v) p = (x, y, z)

u = x/z, v=y/z

Line  Line

Line 

Au + Bv + C = 0

Plane

Recovering 3D Direction

• Let’s represent the 3D line parametrically, i.e. as the set of p0+td for all t, where d is the direction.

• The 9 parallel lines on the board have the same d but different p0.

• For all t: L•(p0+td) = L•p0 + t L•d = 0 • Since p0 is on the line, L•p0 = 0.

• Therefore, L•d = 0, i.e. Axd + Byd + Czd = 0

2D 3D

p = (u, v) p = (x, y, z)

u = x/z, v=y/z

Line  Line

Line 

Au + Bv + C = 0

Plane

Ax + By + Cz = 0 L•p = 0

Common Intersection  Parallel Lines

Finding The Vanishing Point

• Want to find a group of 9 2D lines with a common intersection point

• For two lines A1u+B1v+C1 = 0 and A2u+B2v+C2

= 0, intersection point is on both lines, therefore satisfies both linear equations:

• In reality, only

Representing the Vanishing Point

• Problem: If camera is perpendicular to board, 2D lines are parallel  no solution

• Problem: If camera is almost perpendicular to board, small error in line angle make for big changes in

intersection location

Desired Metric

• Sensitivity Analysis for Intersection Point:

• Of the form u’/w, v’/w. If -1 ≤ A,B,C ≤ 1, then -2 ≤ u’, v’, w ≤ 2

• So, the only way for the intersection point to be large is if w is small; the right metric is roughly

Homogeneous Coordinates

• Introduced by August Ferdinand Möbius • An example of projective geometry

• Represent a 2D point using 3 numbers

• The point (u, v, w) corresponds to u/w, v/w

• Formula for perspective projection means any

3D point is already the homogeneous coordinate of its 2D projection

Homogeneous Coordinates

(u, v, 1)

Clustering: Computer Vision In The

Nineties

• 60s and 70s: Promise of human

equivalence right around the corner

• 80s: Backlash against AI

– Like an “internet startup” now

• 90s: Extensions of existing engineering

techniques

– Applied statistics: Bayesian Networks

2D to 3D: Distance

• How do we get the distance to the board? From the spacing between lines.

Distance To Board

• Let c = (0, 0, zc) be the

point were the z axis hits the board

• For each line in group 1, we find the 3D

perpendicular distance from c to the line, along d2

• These should be equally spaced in 3D

Distance To Board

• Let Li = (Ai, Bi, Ci) be the ith line in group 1.

• We want to find ti such that the point c + ti d2 is

on Li, i.e.

• Li•(c + ti d2) = 0

• Li•c + ti Li•d2 = 0

• ti = - Li•c / Li•d2 = - Cizc / Li•d2

• zc is the only unknown, so put that on the left

• Let si = ti/zc = -Ci / Li•d2

Distance To Board

• So, given the s

_i

, we need to find t

₀

and z

_c

such that:

• s

i

= t

i

/z

c

= (i + t

0

) / z

c

, i = 0…8

• But, we have outliers & occasional

omission

Robust Estimation

• Many existing parameter estimation algorithms optimize a continuous function

– Sometimes there’s a closed form (e.g. MLE of center of Gaussian is just the sample mean)

– Sometimes it’s something more iterative (e.g. Newton-Rhapson)

• However, these are usually sensitive to outliers

– Data cleaning is often a big part of the analysis

Robust Estimation

• Find distance (and offset) to maximize

score:

– Involves evaluating on a fine grid