smr_defense.ppt 3624KB Jun 23 2011 12:31:06 PM

(1)

Real-time Acquisition and

Rendering of Large 3D

Models

Szymon Rusinkiewicz

(2)

Computer Graphics Pipeline

• _{Human time = expensive}_{Human time = expensive} • Sensors = cheap_{Sensors = cheap}

– Computer graphics increasingly relies onComputer graphics increasingly relies on measurements of the real world

measurements of the real world Rendering Rendering Rendering Rendering Shape Shape Shape Shape Lighting Lighting and and Reflectance Reflectance Lighting Lighting and and Reflectance Reflectance Motion Motion Motion Motion Shape

(3)

3D Scanning Applications

• _Computer_Computer graphics graphics • _Product_Product

inspection inspection • Robot _Robot

navigation navigation • _As-built_As-built

floorplans floorplans

• _{Product design}_{Product design} • _Archaeology_Archaeology

(4)

The Digital Michelangelo

Project

• _{Push state of the art in range}_{Push state of the art in range} scanning and demonstrate

scanning and demonstrate

applications in art and art history applications in art and art history

Working in Working in the museum

the museum Scanning Scanning geometry

geometry Scanning Scanning

(5)

Traditional Range Scanning

Pipeline

• _{High-quality, robust pipeline for}_{High-quality, robust pipeline for}

producing 3D models: producing 3D models:

– ScanScan object with laser triangulation object with laser triangulation

scanner: many views from different angles scanner: many views from different angles

– AlignAlign pieces into single coordinate frame: pieces into single coordinate frame: initial manual alignment, refined with ICP initial manual alignment, refined with ICP

– MergeMerge overlapping regions: compute overlapping regions: compute

“average” surface using VRIP [Curless & “average” surface using VRIP [Curless &

Levoy 96] Levoy 96]

(6)

3D Scan of David:

Statistics

• Over 5 meters tall_{Over 5 meters tall}

• 1/4 mm resolution_{1/4 mm resolution}

• 22 people_{22 people}

• 30 nights of scanning_{30 nights of scanning}

• Efficiency max : min = 8 : 1_{Efficiency max : min = 8 : 1}

– Needed view planningNeeded view planning

• Weight of gantry: 800 kg_{Weight of gantry: 800 kg}

• Putting model together:_{Putting model together:} 1000+ man-hours and 1000+ man-hours and counting

(7)

New 3D Scanning Pipeline

• Need for a fast, inexpensive,_{Need for a fast, inexpensive,}

easy-to-use 3D scanning system

• Wave a (small, rigid) _{Wave a (small, rigid)}

object by hand in front of

the scanner

• Automatically align data _{Automatically align data}

as

it is acquired

• _{Let user see partial}_{Let user see partial}

model as it

is being built – fill holes

Real-Time Model Acquisition

(8)

Real-Time 3D Model

Acquisition

• _{Prototype real-time model}_{Prototype real-time model} acquisition system

acquisition system

– 3D scanning of moving objects3D scanning of moving objects

– Fast alignmentFast alignment

(9)

Applications of Easy-to-Use

3D Model Acquisition

• _Advertising_Advertising

• _{More capabilities in Photoshop}_{More capabilities in Photoshop} • _{Movie sets}_{Movie sets}

(10)

3D Scanning Technologies

• _{Contact-based:}_{Contact-based:}_{touch probes}_{touch probes}

• Passive:_Passive: shape from stereo, motion, shape from stereo, motion, shading

shading

• Active:_Active: time-of-flight, defocus, time-of-flight, defocus,

photometric stereo, triangulation photometric stereo, triangulation

• Triangulation systems are _{Triangulation systems are}

inexpensive, robust, and flexible inexpensive, robust, and flexible

– Take advantage of trends in DLP Take advantage of trends in DLP projectors

(11)

Laser Triangulation

• _{Project laser stripe onto object}_{Project laser stripe onto object}

Object

Laser

Camera Camera Camera

(12)

Camera Camera Camera

Camera

Laser Triangulation

• _{Depth from ray-plane triangulation}_{Depth from ray-plane triangulation}

Object

Laser

(x,y)

(13)

Triangulation

• _{Faster acquisition: project}_{Faster acquisition: project} multiple stripes

multiple stripes

• _{Correspondence problem: which}_{Correspondence problem: which} stripe

stripe

(14)

Triangulation

Slow, robust Fast, fragile

Multi-stripe Multi-frame

(15)

Time-Coded Light Patterns

• _{Assign each stripe a unique}_{Assign each stripe a unique}

illumination code

over time [Posdamer 82]

Space Space Time

(16)

Gray-Code Patterns

• _{To minimize effects of quantization error:}_{To minimize effects of quantization error:}

each point may be a boundary only once

Space Space Time

(17)

Structured-Light

Assumptions

• _{Structured-light systems make certain}_{Structured-light systems make certain}

assumptions about the scene:

assumptions about the scene: • Spatial_Spatial continuity assumption: continuity assumption:

– Assume scene is one objectAssume scene is one object

– Project a grid, pattern of dots, etc.Project a grid, pattern of dots, etc. • _Temporal_Temporal_{continuity assumption:}_{continuity assumption:}

– Assume scene is staticAssume scene is static

(18)

Codes for Moving Scenes

• _{We make a different assumption:}_{We make a different assumption:} – Object may moveObject may move

– Velocity low enough to permit Velocity low enough to permit tracking

tracking

(19)

Illumination history = (WB),(BW),(WB)

Illumination history = (WB),(BW),(WB) Illumination history = (WB),(BW),(WB) Illumination history = (WB),(BW),(WB)

Code

Code Code Code

Codes for Moving Scenes

• Code stripe _{Code stripe}boundariesboundaries instead of stripes

instead of stripes

• Perform frame-to-frame_{Perform frame-to-frame} tracking of corresponding tracking of corresponding

boundaries boundaries

– Propagate illumination Propagate illumination history

history

[Hall-Holt & Rusinkiewicz, ICCV [Hall-Holt & Rusinkiewicz, ICCV 2001]

(20)

New Scanning Pipeline

Project

Code

Code Project Project Code

Code CaptureCaptureImages_Images Capture Capture Images

Images Boundaries_BoundariesFindFind Find Find Boundaries

Boundaries Boundaries_BoundariesMatchMatch Match Match Boundaries

Boundaries DecodeDecodeDecodeDecode ComputeComputeRange_Range Compute Compute

(21)

Designing a Code

• _{Biggest problem is ghosts – WW or}_{Biggest problem is ghosts – WW or} BB “boundaries” that can’t be

BB “boundaries” that can’t be seen directly seen directly Project Project Code Code Project Project Code

Code CaptureCaptureImages_Images Capture

Capture

Images

Images Boundaries_BoundariesFindFind Find

Find

Boundaries

Boundaries Boundaries_BoundariesMatchMatch Match

Match

Boundaries

Boundaries DecodeDecodeDecodeDecode ComputeComputeRange_Range Compute

Compute

Range

(22)

Designing a Code

• _{Design a code to make tracking}_{Design a code to make tracking} possible:

possible:

– Do not allow two Do not allow two spatiallyspatially adjacent adjacent ghosts

ghosts

– Do not allow two Do not allow two temporallytemporally adjacent adjacent ghosts

(23)

Designing a Code

• Graph (for 4 frames):_{Graph (for 4 frames):}

– Edges: boundaries (over time)Edges: boundaries (over time) – Nodes: stripes (over time)Nodes: stripes (over time)

0011

1110

1011

1011 01100110

0100

0100 10011001 0001

0001 11001100 0000

0000 11011101

1010

1010 01110111

1000

0101

0101 11111111 00100010

Space Space Time

(24)

Designing a Code

• Graph (for 4 frames):_{Graph (for 4 frames):}

• Path with alternating colors:_{Path with alternating colors:}

55 edges in graph

55 edges in graph 

maximal-length traversal has 110 boundaries (111

stripes)

– Edges: boundaries (over time)Edges: boundaries (over time) Boundary visible at even times

Boundary visible at even times

Boundary visible at odd times

– Nodes: stripes (over time)Nodes: stripes (over time)

0011

1110

1011

1011 01100110

0100

0100 10011001 0001

0001 11001100 0000

0000 11011101

1010

1010 01110111

1000

0101

(25)

Image Capture

• Standard video camera: fields at _{Standard video camera: fields at} 60 Hz

60 Hz

• _{Genlock camera to projector}_{Genlock camera to projector}

Project Project Code Code Project Project Code

Code CaptureCaptureImages_Images Capture Capture Images

Find

Boundaries

Match

Boundaries

Compute

Range

(26)

Finding Boundaries

Capture

Images

Images Boundaries_BoundariesFindFind Find Find Boundaries

Match

Boundaries

Compute

Range

• _{Standard edge detection problem}_{Standard edge detection problem} • Current solution: find minima and _{Current solution: find minima and}

maxima of intensity, boundary is maxima of intensity, boundary is

(27)

Matching Stripe Boundaries

Capture

Images

Find

Boundaries

Boundaries Boundaries_BoundariesMatchMatch Match Match Boundaries

Compute

Range

• _{Even if number of ghosts is}_{Even if number of ghosts is}

minimized, matching is not easy minimized, matching is not easy

?

(28)

Matching Stripe Boundaries

• _{Resolve ambiguity by constraining}_{Resolve ambiguity by constraining} maximum stripe velocity

maximum stripe velocity

• _{Could accommodate higher speeds}_{Could accommodate higher speeds} by estimating velocities

by estimating velocities

• Could take advantage of methods _{Could take advantage of methods} in

in

tracking literature (e.g., Kalman tracking literature (e.g., Kalman

(29)

Decoding Boundaries

Capture

Images

Find

Boundaries

Match

Boundaries

Compute

Range

• _{Propagate illumination history}_{Propagate illumination history}

• Table lookup based on illumination _{Table lookup based on illumination} history and position in four-frame history and position in four-frame

sequence sequence

– Once a stripe has been tracked for at least Once a stripe has been tracked for at least four frames,

four frames,

it contributes useful data on

it contributes useful data on everyevery subsequent frame

(30)

Computing 3D Position

• _{Ray-plane intersection}_{Ray-plane intersection} • _{Requires calibration of:}_{Requires calibration of:}

– Camera, projector intrinsicsCamera, projector intrinsics

– Relative position and orientationRelative position and orientation

Capture

Images

Find

Boundaries

Match

Boundaries

Boundaries DecodeDecodeDecodeDecode ComputeComputeRange_Range Compute Compute

(31)

Results

Video Video frames frames

Stripe Stripe boundaries boundaries

unknow

n

known

ghosts

(32)

Results

• _{Single range image of}_{Single range image of}_moving_moving object

object

Gray codes, no tracking

Gray codes, no tracking Boundary codes and trackingBoundary codes and tracking

Top View

(33)

Aligning 3D Data

• _{This range scanner can be used}_{This range scanner can be used} for any moving objects

for any moving objects

• _For_For_{rigid objects}_{rigid objects}_{, range images}_{, range images} can be aligned to each other as can be aligned to each other as

(34)

Aligning 3D Data

• _{If correct correspondences are}_{If correct correspondences are} known,

known,

it is possible to find correct it is possible to find correct

(35)

Aligning 3D Data

• _{How to find corresponding points?}_{How to find corresponding points?} • _{Previous systems based on user}_{Previous systems based on user}

input, input,

feature matching, surface feature matching, surface

(36)

Aligning 3D Data

• _{Alternative: assume}_{Alternative: assume}_closest_closest_points_points correspond to each other, compute correspond to each other, compute

(37)

Aligning 3D Data

• _…_…_{and iterate to find alignment}_{and iterate to find alignment}

– Iterated Closest Points (ICP) [Besl & Iterated Closest Points (ICP) [Besl & McKay 92]

McKay 92]

• Converges if starting position _{Converges if starting position} “close enough“

(38)

ICP Variants

• Classic ICP algorithm not real-timeClassic ICP algorithm not real-time

• To improve speed: examine stages of ICP and evaluate proposed variantsTo improve speed: examine stages of ICP and evaluate proposed variants

[Rusinkiewicz & Levoy, 3DIM 2001]

[Rusinkiewicz & Levoy, 3DIM 2001]1.1. SelectingSelecting source points (from one or both meshes) source points (from one or both meshes) 2.

2. MatchingMatching to points in the other mesh to points in the other mesh 3.

3. WeightingWeighting the correspondences the correspondences 4.

4. RejectingRejecting certain (outlier) point pairs certain (outlier) point pairs 5.

5. Assigning an Assigning an error metricerror metric to the current transform to the current transform 6.

(39)

ICP Variant –

Point-to-Plane Error Metric

• _{Using point-to-plane distance}_{Using point-to-plane distance}

instead of point-to-point lets flat instead of point-to-point lets flat

regions slide along each other regions slide along each other

(40)

Finding Corresponding

Points

• _{Finding closest point is most expensive}_{Finding closest point is most expensive}

stage of ICP

– Brute force search – O(n)Brute force search – O(n)

– Spatial data structure (e.g., k-d tree) – Spatial data structure (e.g., k-d tree) – O(log n)

O(log n)

– Voxel grid – O(1), but large constant, slow Voxel grid – O(1), but large constant, slow preprocessing

(41)

Finding Corresponding

Points

• _{For range images, simply project point}_{For range images, simply project point}

[Blais 95]

– Constant-time, fastConstant-time, fast

– Does not require precomputing a spatial data Does not require precomputing a spatial data structure

(42)

High-Speed ICP Algorithm

• _{ICP algorithm with projection-}_{ICP algorithm with}

projection-based correspondences, based correspondences,

point-to-plane matching plane matching

can align meshes in a few tens of can align meshes in a few tens of

ms. ms.

(43)

Anchor Scans

• _{Alignment of consecutive scans}_{Alignment of consecutive scans} leads to accumulation of ICP

leads to accumulation of ICP errors

errors

• Alternative: align all scans to an _{Alternative: align all scans to an} “anchor” scan, only switch anchor “anchor” scan, only switch anchor

when overlap low when overlap low

• _{Given anchor scans, restart after}_{Given anchor scans, restart after} failed ICP becomes easier

(44)

Merging and Rendering

• _{Goal: visualize the model well enough}_{Goal: visualize the model well enough}

to be able to see holes to be able to see holes

• _{Cannot display all the scanned data –}_{Cannot display all the scanned data –}

accumulates linearly with time accumulates linearly with time

• _{Standard high-quality merging}_{Standard high-quality merging}

methods: methods:

(45)

Merging and Rendering

• _{Real-time incremental merging}_{Real-time incremental merging} and rendering:

and rendering:

– Quantize samples to a 3D gridQuantize samples to a 3D grid

– Maintain average normal of all pointsMaintain average normal of all points at a grid cell

at a grid cell

– Point (splat) renderingPoint (splat) rendering

– Can be made hierarchical to conserve Can be made hierarchical to conserve memory

(46)

Photograph

(47)

Real-time Scanning Demo

(48)

Postprocessing

• _{Goal of real-time display is to let}_{Goal of real-time display is to let} user evaluate coverage, fill holes user evaluate coverage, fill holes

– Quality/speed tradeoff Quality/speed tradeoff

• Offline postprocessing for high-_{Offline postprocessing for} high-quality models

(49)

Merged Result

Photograph

Photograph Aligned scansAligned scans

(50)

Future Work

• Technological improvements:_{Technological improvements:}

– Use full resolution of projectorUse full resolution of projector

– Higher-resolution camerasHigher-resolution cameras

– Ideas from design of single-stripe 3D scannersIdeas from design of single-stripe 3D scanners

• _{Pipeline improvements:}_{Pipeline improvements:}

– Better detection of failed alignmentBetter detection of failed alignment

– Better handling of object texture – combine Better handling of object texture – combine with stereo?

with stereo?

– Global registration to eliminate driftGlobal registration to eliminate drift

– More sophisticated mergingMore sophisticated merging

(51)

Future Work

• _{Faster scanning}_{Faster scanning}

– Better stripe boundary matchingBetter stripe boundary matching

– Multiple cameras, projectorsMultiple cameras, projectors

– High-speed camerasHigh-speed cameras

• _{Application in different contexts}_{Application in different contexts}

– Small, hand-heldSmall, hand-held

– Cart- or shoulder-mounted for digitizing _{Cart- or shoulder-mounted for digitizing}

rooms

(52)

Rendering of Large Models

• Range scanners increasingly capable of _{Range scanners increasingly capable of} producing very large models

producing very large models

– DMich models are 100 million to 1 billion samplesDMich models are 100 million to 1 billion samples

• _Challenge:_Challenge:_{how to allow viewing in real time}_{how to allow viewing in real time}

– Fast startup, progressive loadingFast startup, progressive loading

• Traditional answer:_{Traditional answer:} triangle meshes, triangle meshes, simplification, hardware-accelerated simplification, hardware-accelerated

rendering rendering

– Impractical for such large modelsImpractical for such large models

• Alternative:_Alternative: revisit basic data structure revisit basic data structure

QSplat [Rusinkiewicz & Levoy, SIGGRAPH 00]

(53)

QSplat

• _{Key observation: a single}_{Key observation: a single}

bounding sphere hierarchy can be bounding sphere hierarchy can be

used for used for

– Hierarchical frustum and backface Hierarchical frustum and backface culling

culling

– Level of detail controlLevel of detail control

(54)

QSplat Node Structure

Position

and

Radius

Tree

Structure

Structure NormalNormal

Width of

Cone of

Normals

Color

(Optional)

13 bits

13 bits 3 bits3 bits 14 bits14 bits 2 bits2 bits 16 bits16 bits

6 bytes

(55)

QSplat Node Structure

• Position and radius encoded _{Position and radius encoded}

relative to parent node

– Hierarchical coding vs. delta Hierarchical coding vs. delta coding along a path for vertex

coding along a path for vertex

positions positions Position Position and and Radius Radius Tree Tree Structure

Width of Width of Cone of Cone of Normals Normals Color Color (Optional) (Optional) 13 bits

Center Offset Center Offset

Radius Ratio

(56)

QSplat Node Structure

Position

and

Radius

Tree

Structure

Width of

Cone of

Normals

Color

(Optional)

13 bits

Uncompressed

(57)

Position Position and and Radius Radius Tree Tree Structure

Delta Coding

[Deering 96]

(58)

QSplat Node Structure

Position Position and and Radius Radius Tree Tree Structure

Hierarchical

Coding

(59)

QSplat Rendering Algorithm

• _{Traverse hierarchy recursively}_{Traverse hierarchy recursively}

if

if (node not visible)(node not visible)

Skip this branch

else if

else if (leaf node) (leaf node)

Draw a splat

else if

else if (size on screen < (size on screen < threshold)

threshold)

Draw a splat

else

Traverse children

if

if (node not visible)(node not visible)

Skip this branch

else if

else if (leaf node) (leaf node)

Draw a splat

else if

else if (size on screen < (size on screen < threshold)

threshold)

Draw a splat

else

Traverse children

Hierarchical frustum /

backface culling

Point rendering

Adjusted to maintain

desired frame rate

Level of detail

control

(60)

Demo – St. Matthew

• _{3D scan of 2.7 meter}_{3D scan of 2.7 meter} statue at 0.25 mm

statue at 0.25 mm • _{102,868,637 points}_{102,868,637 points} • File size: 644 MB_{File size: 644 MB}

• Preprocessing time: _{Preprocessing time:} 1 hour

(61)

Future Work

• Splats as primitive_{Splats as primitive}

– Unify rendering of meshes, volumes, point Unify rendering of meshes, volumes, point clouds

clouds

– Compatible with shading after rasterizationCompatible with shading after rasterization

– Hybrid point/polygon systemsHybrid point/polygon systems

• High-level visibility / LOD frameworks_{High-level visibility / LOD frameworks}

– Store different kinds of data at each node: Store different kinds of data at each node: alpha, BRDF, scattering function, etc.

alpha, BRDF, scattering function, etc.

– Potentially could be used to unify image-Potentially could be used to unify image-based-rendering (IBR) techniques

(62)

Contributions

• _{Real-time 3D model acquisition system}_{Real-time 3D model acquisition system}

– Video-rate 3D scanner for moving objectsVideo-rate 3D scanner for moving objects

– Analysis of ICP variants; real-time algorithmAnalysis of ICP variants; real-time algorithm

– Real-time merging and renderingReal-time merging and rendering

– Allows user to see model and fill holesAllows user to see model and fill holes

• _{QSplat: interactive rendering of large 3D}_{QSplat: interactive rendering of large 3D}

meshes

– Single data structure used for visibility culling,Single data structure used for visibility culling, level-of-detail control, point rendering,

level-of-detail control, point rendering,

compression

(63)

Acknowledgments

• Olaf Hall-Holt_{Olaf Hall-Holt}

• Lucas Pereira_{Lucas Pereira}

• The Original DMich Gang: Dave Koller, Sean _{The Original DMich Gang: Dave Koller, Sean}

Anderson, James Davis, Kari Pulli, Matt Ginzton, Jon

Shade

• DMich, the next generation: Gary King, Steve _{DMich, the next generation: Gary King, Steve}

Marschner

• Graphics lab_{Graphics lab}

• Advisor: Marc Levoy_{Advisor: Marc Levoy}

• Committee: Pat Hanrahan, Leo Guibas, Mark _{Committee: Pat Hanrahan, Leo Guibas, Mark}

Horowitz, Bernd Girod

• Family, friends_{Family, friends}