UNIVERSITI TEKNIKAL MALAYSIA MELAKA
MOTION TRACKING SYSTEM USING COMPUTER VISION
WITH OPENCV SOURCE SOFTWARE
This report in accordance with requirement of the Universiti Teknikal Malaysia Melaka
(UTeM) for Bachelor Degree of Manufacturing Engineering (Robotic and Automation)
with Honours.
by
WIRADANI BIN MD ALI
FACULTY OF MANUFACTURING ENGINEERING
UNIVERSITI TEKNIKAL MALAYSIA MELAKA
BORANG PENGESAHAN STATUS LAPORAN PSM
TAJUK:
‘Motion Tracking System Using Computer Vision With OpenCV Source Software’
SESI PENGAJIAN:
2009/2010 Semester 2
Saya WIRADANI BIN MD ALI
mengaku membenarkan laporan PSM ini disimpan di Perpustakaan Universiti Teknikal Malaysia Melaka (UTeM) dengan syarat-syarat kegunaan seperti berikut:
1. Laporan PSM / tesis adalah hak milik Universiti Teknikal Malaysia Melaka dan
penulis.
2. Perpustakaan Universiti Teknikal Malaysia Melaka dibenarkan membuat salinan
untuk tujuan pengajian sahaja dengan izin penulis.
3. Perpustakaan dibenarkan membuat salinan laporan PSM / tesis ini sebagai bahan
pertukaran antara institusi pengajian tinggi.
4. *Sila tandakan (√)
SULIT
TERHAD
⁄ TIDAK TERHAD
(Mengandungi maklumat yang berdarjah keselamatan atau
kepentingan Malaysia yang termaktub di dalam AKTA RAHSIA RASMI 1972)
(Mengandungi maklumat TERHAD yang telah ditentukan oleh organisasi/badan di mana penyelidikan dijalankan)
APPROVAL
This report is submitted to the Faculty of Manufacturing Engineering of UTeM as a
partial fulfillment of the requirements for the degree of Bachelor of Manufacturing
Engineering (Robotic and Automation) with Honours. The members of the
supervisory committee are as follow:
(Signature of Supervisor)
DECLARATION
I hereby declared this report entitled “Motion Tracking System Using Computer Vision With OpenCV Source Software” is the result of my own research except as cited in the references.
Signature : ………..
Author’s Name : ………..
i
ABSTRACT
Nowadays motion tracking widely used for some field, especially autonomous field. The
motion tracking also has been used for securities. At the Oversea for example United
States of America use the motion tracking in army usage. This project purpose is to do
the motion tracking using the OpenCV software. The motion tracking is the camera will
detect and track the object movement. This project will include the research for the
related previous work as a reference to done the motion tracking method. The related
previous work will cover in the literature review, the method used are research based on
the scopes of this project. This scopes project is only cover the color based, optical flow
and feature extraction image processing technique. The methodology chooses for this
motion tracking use is edge detection under the optical flow technique and the color
based will used the segmentation technique. The methodology chooses is based on the
research has been done and the techniques is efficiencies and many of the work done by
ii
ABSTRAK
Pada era sekarang “motion tracking” ini digunakan secara meluas untuk beberapa
bidang, terutama bidang autonomi. “motion tracking” juga telah digunakan untuk system
keselamatan. Di luar Negara misalnya Amerika Syarikat telah menggunakan “motion
tracking” untuk kegunaan tentera. Objektif projek ini adalah untuk melakukan “motion
tracking” perisian menggunakan OpenCV. “motion tracking” kamera akan mengesan
dan menjejaki pergerakan objek. Projek ini merangkumi kajian sebelumnya yang
berkaitan dengan tajuk sebagai rujukan untuk melakukan “motion tracking” cara kerja.
Hasil kerja sebelumnya yang berkaitan akan merangkumi dalam kajian lapangan, cara
kerja dan penelitian yang digunakan adalah berdasarkan pada takat projek ini. Skop
projek ini hanya merangkumi berasaskan melalui cara warna, optik aliran dan ekstraksi
ciri pengolahan gambar teknik. Cara kerja dipilh untuk menggunakan “motion tracking”
ini adalah deteksi tepi aliran di bawah teknik optik dan warna yang didasarkan akan
menggunakan teknik segmentasi. Memilih cara kerja berdasarkan kajian yang telah
dilakukan dan teknik yg digunakan berpandukan daripada kecekapan dan daripada hasil
iii
DEDICATION
Specially dedicated to my beloved father Md Ali Sukardi and my mother Zaitom Abdul
Hamidwho are very concern, understanding, patient, and supporting. Thanks for
everything to my supervisor Mr. Shariman Bin Abdullah for his constructive guidance,
encouragement and patient in fulfilling our aspiration in completing this project, to my
sister, and all my friends. I also would like to say thanks for everything. The work and
iv
ACKNOWLEDGEMENT
I would like thank to my supervisor Mr. Shariman Abdullah for helping, guide me and
encourage me to finish this motion tracking project. Without guide from my supervisor I
could not finish this project successfully. Then I would like to thank to my entire friend
that give me a guide to finish the report with the correct format and teach me how to
write in English with correct grammar. Not forget I would like to thank to my parent that
not tired give me encouragement to finish my studies and always be my backbone of my
successful. Lastly I would like to thank to all that directly or indirectly involve finishing
v
1.5 Research Objectives 3
1.6 Research Scopes 3
2.2.1 Definition to Motion Tracking 7
2.3 Motion Tracking Techniques 9
2.3.1 Optical Flow Based 10
2.3.1.1 Tracking with Background Subtraction 12
vi
2.3.2.1 Color Based Motion Tracking Technique (The RGB Space) 17
2.3.2.2 Color Based Motion Tracking Technique (The Greyscale Space) 18
2.3.2.3 Color Based Motion Tracking Technique (The Hue, Saturation and
Lightness Space)
18
2.3.2.4 Color Based Motion Tracking Technique (Chromatic Mapping) 20
2.4 Feature Extraction and Image Processing 21
2.4.1 Image Filtering 21
2.5 Previous Related Work 23
2.5.1 Color, texture, and motion in level set based segmentation and
tracking
23
2.5.2 Object tracking using SIFT features and mean shift 28
2.5.3 Reliable and Fast Human Body Tracking under Information
Deficiency
34
2.5.4 Detection and Tracking of Human Subjects 37
2.5.5 Region-based Pose Tracking 42
2.5.6 Statistical Cue Estimation for Model-Based Shape and Motion 46
2.5.7 A Hybrid Color-Based Foreground Object Detection Method for
Automated Marine Surveillance
51
2.5.8 Tracking a Tennis Ball Using Image Processing Techniques 55
2.6 Open Computer Vision 62
2.6.1 Open Computer Vision Implementation 62
3 METHODOLOGY 65
3.0 Background 65
3.1 Flow Chart of the Process 66
3.2 Motion Tracking Methodology 67
3.3 Optical Flow 67
3.3.1 Background Subtraction 68
3.3.1.1 Edge Detection 69
vii
3.4.1 Red, Green, Blue (RGB) 70
3.4.1.1 Segmentation 71
3.5 Mean Shift (camshift) 72
3.6 Motion Tracking Technique Combination 73
4 DEVELOPMENT 75
5.1 Moving Slowly Toward and Away From the Camera 87
5.2 Swaying Side to Side 88
5.3 Rotational 89
5.4 All Over the Camera with Back Projection 90
6 CONCLUSION AND RECOMMENDATION 91
6.0 Background 91
6.1 Conclusion 91
6.2 Recommendation 92
viii
LIST OF TABLES
1.1 Gantt Chart for Motion Tracking Project PSM 1 4
1.2 Gantt Chart for Motion Tracking Project PSM 2 5
ix
LIST OF FIGURES
2.1 The Class of the Motion Tracking 7
2.2 The Motion Tracking of Face Detection 8
2.3 The Object (face) Has Been Identified and Recognized 9
2.4 The Original Image 16
2.5 The Segmented Target Object 17
2.4 The RGB Cube 17
2.11 Segmentation Results for Synthetic Texture Images 25
2.12 Segmentation Results with Some Real-World Images Using Texture and
Color Information
25
2.13 Benefits of Using Joint Color and Texture Information 26
2.14 Tracking of Small Objects in a Cutout of the Traffic Sequence 26
2.15 Tracking of Three Overlapping Balls 27
2.16 Tracking of Three Players in a Soccer Sequence 27
2.17 Test Sequences Used in Current Evaluation 29
2.18 Tracking Comparison of the Classical Mean Shift 31
2.19 Tracking Comparison of the Classical Mean Shift 32
2.20 Performance Comparisons of Classical Mean Shift 33
2.21 Output Images 35
2.22 Face Detection Result 36
2.23 The Image of Walking Woman has been Change to Gray Scale 38
2.24 Subtracting Background Frame from Current Frame 39
x
2.26 Center-of-Mass The first Frame with the Center-of-Mass Displayed 41
2.27 Tracking Rectangle The same Frame with the Rectangle Visualization 42
2.28 Problems Often Occurring in Variation Segmentation Algorithms 43
2.29 The Giraffe Images has been Detect 44
2.30 The Images of Box 45
2.31 An Example of the Results of a Canny Edge Detection and the Distance
Field of the Edges
47
2.32 Sequence of the Motion Tracking 49
2.32 The Sequence of Motion Tracking Using the Canon Camera with Cue
Algorithm Method
51
2.32 Block Diagram of the Proposed Foreground Segmentation System 52
2.33 The Experiment Performed on a Video Containing Typical Marine
Surveillance Footage
53
2.34 Results Obtained for Representative Frames 54
2.35 The Sample of Background Image has been Done the Edges 56
2.36 The Foreground Object Detection Algorithm 56
2.37 The Sequence of the Object Detection 57
2.38 The Illustrated of Segmentation Approach 58
2.39 The Sequence Image Tennis Ball Detection Using the Colour
Segmentation Approach
59
2.40 The Image of Tennis Ball with Shape Recognition Approach 60
2.41 The Illustrated of the Classifier Creation 61
2.42 Image of Tennis Ball Detection Using the Classifier Creation 61
2.43 The Example of the Threshold Processing 63
3.1 Methodology Motion Tracking Flow Chart 67
3.2 The Example of the Background Subtraction 69
3.3 The Result of the Edge Detection 70
3.4 The RGB Technique 72
xi
3.6 An example of the camshift face tracking 73
3.7 Example of the Visual Studio C++ 2008 Express Edition with C++
xii
LIST OF ABBREVIATIONS
2D - Two Dimension
3D - Three Dimension
EKF - Extended Kalman filter
fps - frames per second
HS - Hue and Saturation
HSL - hue, saturation, and lightness
HSV - hue, saturation, and value
IEKF - Iterative Extended Kalman Filter
IIR - Impulse Response
OpenCV - Open Computer Vision
RGB - Red, Blue, and Green
1
CHAPTER 1
INTRODUCTION
1.1Background
In this chapter will describes and simple briefly about the motion tracking and OpenCV
definition. In this chapter also include the problem statement of the motion tracking,
objective of this project, research scopes, and lastly important of the research.
1.2Motion Tracking
What is motion tracking? Motion tracking is the process where the vision of image or
videos detected. The movement on the camera vision range will be tracking. Many
things can be used of motion tracking functional. Below statement is the definition of the
motion tracking.
Motion tracking also used in media sectors. According to J. Sturman, 2001, motion
tracking is the recording of human body movement for analysis and playback. The
information of the tracking can be as simple as the position of the body in space or as
complex as the deformations of the face and muscle masses. Motion tracking for
computer character animation involves the mapping of human motion onto the motion of
a computer character. The mapping can be direct, such as human arm motion controlling
a character's arm motion, or indirect, such as human hand and finger patterns controlling
2 1.3Open Computer Vision
The history, in 1999 Open Computer Vision (OpenCV) was started at Intel create by
Gary Bradski by purposes of accelerating research and commercial applications of
computer vision. OpenCV is one of kind software installed to create, to program, and to
run with a device to process an image or videos. OpenCV can be viewed as signal
processing applied to higher dimensions of image and videos. OpenCV library usually
used the C and C++ to programming the database. The OpenCV is optimized and intend
for real time applications.
1.4Problem Statement
The motion tracking is not completely used in the autonomous robotic technologies. The
motion tracking other problem is to implement to the future technologies where it can be
detected and tracking the motion of the object or human. Then motion tracking
methodologies applied to the real time world also is the problem, how the motion
tacking can be done with the right method and can be implementing to the real time
functionality. Next, the motion tracking functionality, what must be detected, how wants
to detected, for a moving object which is, the robot car, color detection and thus
recognition of the color object is of a more flexible concept where no database is
inserted instead the object is recognized through a predefined color. On the same basic,
they hold the same idea that is, to recognize a moving object.
1.5Research Objectives
The objective of the project is to track and detect the motion or movement of object
selected. Means the object selected must be followed to anywhere of the movement of
selected object as long as the selected object in webcam viewer range parameters.
3
1.6Research Scopes
The motion tracking have many type and ways to be done. However for this project the
scope of project is to do the motion tracking of object based on the colours based
technique, the camshift technique with the histogram and back projection also the
feature extraction image processing technique for the this project of motion tracking
methodology.
1.7Importance of the Research
Research processes is to Programming the motion tracking by using the OpenCV
software. The research will help student to understand about motion tracking. Then
apply to the real time function maybe as a security system to track and detect the
movement of suspicious object. The motion tracking also can be applied in industrial
function where the technologies of motion tracking can be as a helpful system to track
and detect the production or anything. The motion tracking also is used in automation
and robotic sector. Nowadays the motion tracking mostly can be applied to the army
system for defense of country. For example satellite communication to detect the
4 1.8Gantt Chart of the Motion Tracking
Table 1.1: Gantt chart for motion tracking project PSM 1
History
Actual
5
Table 1.2: Gantt chart for motion tracking project PSM 2
History
Actual
6
CHAPTER 2
LITERATURE REVIEW
2.1 Background
This chapter will discuss and review available literature on Motion Tracking using
OpenCV software. The reviews begin with motion tracking. This section will be
including the motion tracking technique and related previous projects. Then, the
OpenCV will be discussed in this chapter.
2.2 Motion Tracking
This title will discuss about the motion tracking definition. The motion tracking
technique also will be describe in this topic. The technique will be divide into a few sub
topic depend on the scope of the motion tracking research. And many more will be
discuss in this topic.
A central thread of computer vision research is the development of algorithm or system
to track the position and orientation of a target object or objects within images or image
sequences. Object tracking, while a simple task for humans is monumentally more
challenging for computer vision systems. Over the years, a vast number of algorithms
7
require such algorithms to track different target in different conditions (Maurin et al.
2005). For example, to guide an autonomous vehicle in a simple or complex
environments (Kia and Arshad 2005, and Asif et al. 2005) or it may be used to track
vehicle for collecting the traffic data from highway scenes (Kastrinaki et al. 2003) or
even to detect human in a surveillance system (Collins et al. 2000a). Tracking may also
be used in robot arm applications either to provide guidance to surgical robot (Ginhoux
et al. 2003, and Zhang and Payandeh 2002) or to select an optimal grasp for picking-up
object (Han and Kuc 1998). General techniques for tracking are independent from any
particular application.
2.2.1 Definition to Motion Tracking
According to the Anonymous, (1991) the motion tracking definition is to track an object
over a sequence of images. Object tracking in general, is a challenging problem.
Difficulties in tracking objects can arise due to abrupt object motion, changing
appearance patterns of the object and the scene, non-rigid object structures, object to
object and object to scene occlusions, and camera motion. Tracking is usually performed
in the context of higher-level applications that require the location or shape of the object
in every frame. Generally there are three types of tracking algorithm first is point
tracking, then kernel tracking and third is silhouette tracking. The type of the motion
tracking is shown in figure below. The object tracking is the general and main title then
is divided into three types.
8
Paulus, (2009) said that the aim of motion tracking is to detect and track moving objects
through a sequence of images. Motion tracking is not only useful for monitoring activity
in public places, but is becoming a key ingredient for further analysis of video imagery.
For instance, information about the location and identity of objects at different points in
time is the basis of detecting unusual object movements or coordinated activities. With
motion tracking, you can track the movement of an object and then apply the tracking
data for that movement to another object such as another layer or an effect control point
to create compositions in which images and effects follow the motion.
Matsuyama, (2008) believe that the motion tracking is a common requirement for many
real world applications, such as video surveillance, games, cultural and medical
applications example like using for motion and behavior study. Simple object can be
detected and tracked using various image features such as color regions, edges, contours,
or texture. In the other hand, complex objects such as human faces require more
sophisticated features to handle the multiple possible instances of the object class.
Statistical methods are a good alternative. A statistical model learns different patterns
related to the object of interest like motion tracking of the different views of human
faces, including good and bad samples. And then the system is able to estimate whether
a region contains an object of interest or not.
Figure 2.2: Show that the motion tracking of face detection. The red box is the box of reorganization
progress where the face have been detect and not identified yet. The green box shows that the object has
