DEVELOPMENT OF A CHILD DETECTION SYSTEM WITH ARTIFICIAL INTELLIGENCE (AI) USING OBJECT DETECTION
METHOD
Lai Suk Na
Bachelor of Engineering with Honours (Mechanical and Manufacturing Engineering)
2018
FACULTY OF ENGINEERING
FYP REPORT SUBMISSION FORM
Name : ___________________________________________ Matric No. : _______________________
Title : _________________________________________________________________________________
_______________________________________________________________________________________
Supervisor : ___________________________________ Program: ________________________________
Please return this form to the Faculty of Engineering office at least TWO WEEKS before your hardbound report is due.
Students are not allowed to print/bind the final report prior to Supervisor’s Approval (Section B).
The Faculty reserves the right to reject your hardbound report should you fail to submit the completed form within the stipulated time.
A. REPORT SUBMISSION (To be completed by student) I wish to submit my FYP report for review and evaluation.
Signature: ___________________________ Date: ______________
\
B. SUPERVISOR’S APPROVAL (To be completed by supervisor)
The student has made necessary amendments and I hereby approve this thesis for binding and submission to the Faculty of Engineering, UNIMAS.
Signature: ___________________________ Date: ______________
Name: ______________________________________________________
Lai Suk Na
Development of a Child Detection System with Artificial Intelligence (AI) using Object Detection Method 47241
Ir Dr David Chua Sing Ngie Mechanical and Manufacturing Engineering
UNIVERSITI MALAYSIA SARAWAK
Grade:
DECLARATION OF ORIGINAL WORK
This declaration is made on the 22nd day of June 2018.
Student’s Declaration:
I, LAI SUK NA, 47241, FACULTY OF ENGINEERING hereby declare that the work entitled, DEVELOPMENT OF A CHILD DETECTION SYSTEM WITH ARTIFICIAL INTELLIGENCE (AI) USING OBJECT DETECTION METHOD is my original work. I have not copied from any other students’
work or from any other sources except where due reference or acknowledgement is made explicitly in the text, nor has any part been written for me by another person.
22nd June 2018
Date submitted Lai Suk Na (47241)
Supervisor’s Declaration:
I, IR DR DAVID CHUA SING NGIE hereby certifies that the work entitled, DEVELOPMENT OF A CHILD DETECTION SYSTEM WITH ARTIFICIAL INTELLIGENCE (AI) USING OBJECT DETECTION METHOD was prepared by the above named student, and was submitted to the FACULTY OF ENGINEERING as a partial fulfillment for the conferment of BACHELOR OF MECHANICAL AND MANUFACTURING ENGINEERING (Hons.), and the aforementioned work, to the best of my knowledge, is the said student’s work
Received for examination by: _ Date:
(Ir Dr David Chua Sing Ngie)
Please tick ()
Final Year Project Report Masters
PhD
22nd June 2018
I declare this Project/Thesis is classified as (Please tick (√)):
CONFIDENTIAL (Contains confidential information under the Official Secret Act 1972)*
RESTRICTED (Contains restricted information as specified by the organisation where research was done)*
OPEN ACCESS
Validation of Project/Thesis
I therefore duly affirmed with free consent and willingness declared that this said Project/Thesis shall be placed officially in the Centre for Academic Information Services with the abide interest and rights as follows:
This Project/Thesis is the sole legal property of Universiti Malaysia Sarawak (UNIMAS).
The Centre for Academic Information Services has the lawful right to make copies for the purpose of academic and research only and not for other purpose.
The Centre for Academic Information Services has the lawful right to digitise the content to for the Local Content Database.
The Centre for Academic Information Services has the lawful right to make copies of the Project/Thesis for academic exchange between Higher Learning Institute.
No dispute or any claim shall arise from the student itself neither third party on this Project/Thesis once it becomes sole property of UNIMAS.
This Project/Thesis or any material, data and information related to it shall not be distributed, published or disclosed to any party by the student except with UNIMAS permission.
Student’s signature Supervisor’s signature:
(22nd June 2018) (22nd June 2018)
Current Address:
NO 82 TAMAN LANDEH JALAN LANDEH 93250 KUCHING SARAWAK
Notes: * If the Project/Thesis is CONFIDENTIAL or RESTRICTED, please attach together as annexure a letter from the organisation with the period and reasons of confidentiality and
restriction.
[The instrument was duly prepared by The Centre for Academic Information Services]
√
APPROVAL SHEET
This project report which entitled “DEVELOPMENT OF A CHILD DETECTION SYSTEM WITH ARTIFICIAL INTELLIGENCE (AI) USING OBJECT DETECTION METHOD” was prepared and submitted by LAI SUK NA (47241) as a partial fulfilment of the requirement for degree of Bachelor of Engineering with Honours in Mechanical and Manufacturing Engineering is hereby read and approved by:
_____________________
Ir Dr David Chua Sing Ngie (Project Supervisor)
____________________
Date
DEVELOPMENT OF A CHILD DETECTION SYSTEM WITH ARTIFICIAL INTELLIGENCE (AI) USING OBJECT DETECTION
METHOD
LAI SUK NA
A dissertation submitted in partial fulfillment of the requirement for the degree of Bachelor of Engineering with Honours (Mechanical and Manufacturing Engineering)
Faculty of Engineering Universiti Malaysia Sarawak
2018
To my beloved family and friends.
i
ACKNOWLEDGEMENTS
First and foremost, I would like to express my deepest gratitude to my supervisor, Ir Dr David Chua Sing Ngie from Mechanical and Manufacturing Engineering Department for his guidance, advice and contribution of ideas throughout this project. I would also like to thank Google Inc for making object detection model and Tensorflow library open source which enable students and researchers to develop in the field of Artificial Intelligence and machine learning. I sincerely thank Google Images for allowing the usage of images for training purpose. Last but not least, I would like to express my heartiest thank to my family and friends for their support and corporation.
With all their accompany this thesis become reality.
ii
ABSTRACT
The issue of children dying due to vehicular heatstroke has raised the public attention. The failure of current vehicular occupant detection devices to identify correctly the occupant as a child had triggered the idea of developing a child detection system using Artificial Intelligence (AI) technology. The usage of Convolutional Neural Network (CNN) had been recognised as an effective way to perform image classification. However, this approach required a significant number of images as training data and substantial time for model training in order to achieve desired results in accuracy. Due to the limitation of abundant dataset, transfer learning was used to accomplish the task. Modern convolutional object detector, SSD Mobilenet v1 trained on Microsoft Common Objects in Context (MS COCO) dataset was used as a starting point of the training process. MS COCO dataset that consisted of a total of 328k images were divided into 91 different categories including dog, person, kite and so on. The trained model was then retrained to classify adults and children instead of persons. At the end of the training, a real-time child detection system was established. The system was able to give different responses to the detection of a child and adult. The responses comprised of visual and audio outputs. Upon detection, a bounding box was drawn on a child or an adult face as visual output. At the same time, the system would trigger the speaker to speak out the statement “child is detected” for successful child detection whereas adult detection would result in the statement of “adult is detected”. Theoretically, the detection system could achieve an overall precision of 0.969. However, the experimental results obtained was able to match up to a precision of 0.883 that resulted in a small error of 8.88%.
iii
ABSTRAK
Isu kanak-kanak maut dalam kenderaan kerana strok haba telah menimbulkan perhatian orang ramai. Kegagalan alat pengesan di pasaran untuk mengenal pasti kehadiran penumpang kenderaan adalah anak dengan tepat telah mencetuskan idea untuk membangunkan sistem pengesanan kanak-kanak menggunakan teknologi Kecerdasan Buatan (AI). Penggunaan rangkaian neural konvolusi (CNN) telah diakui sebagai cara yang berkesan untuk melakukan klasifikasi imej. Walau bagaimanapun, pendekatan ini memerlukan imej yang banyak sebagai data latihan dan masa yang panjang untuk latihan supaya dapat mencapai ketepatan yang dikehendaki. Oleh sebab batasan dataset yang banyak, pembelajaran pemindahan digunakan untuk menyelesaikan tugas. Pengesan objek konvolusi moden, SSD Mobilenet v1 yang dilatih dalam dataset Microsoft Common Objects Context (MS COCO) digunakan sebagai titik permulaan proses latihan.
Dataset MS COCO mengandungi sejumlah 328 ribu imej dibahagikan kepada 91 kategori yang berbeza termasuk anjing, orang, layang-layang dan sebagainya. Model ini dilatih untuk mengklasifikasikan dewasa dan kanak-kanak yang pada asalnya mengenali kanak- kanak dan orang dewasa sebagai orang. Pada akhir latihan, sistem pengesanan masa nyata ditubuhkan. Sistem memberi respons yang berbeza kepada kanak-kanak dan orang dewasa. Maklum balas terdiri daripada output visual dan audio. Secara teorinya, sistem pengesanan dapat mencapai ketepatan keseluruhan 0.969 sedangkan hasil eksperimen memberikan ketepatan 0.883, memberikan kesilapan sebanyak 8.88%.
iv
TABLE OF CONTENTS
Acknowledgements………..……...……… i
Abstract………...………...………. ii
Abstrak.………...… iii
Table of Contents………...…. iv
List of Tables………...……….. vi
List of Figures………....…………...……….. vii
List of Abbreviations………....………...……... ix
Chapter 1 INTRODUCTION 1.1 General Background………...……… 1
1.2 Problem Statement………...…………... 4
1.3 Objectives………...……… 5
1.4 Scope of Research………....……….. 5
Chapter 2 LITERATURE REVIEW 2.1 History of Artificial Intelligence (AI)……… 6
2.2 Machine Learning (ML)………...……... 7
2.2.1 Supervised Learning………....……….. 7
2.2.2 Unsupervised Learning………...………... 11
2.2.3 Reinforcement Learning……… 12
2.3 Deep Learning (DL)……….……... 14
2.3.1 Perceptron……….. 14
2.3.2 Deep Neural Network……… 16
2.3.2.1 Convolutional Neural Network………... 17
2.3.2.2 Recurrent Neural Network……….. 18
2.4 Pretrained Models………...……… 18
2.5 Related Works………...………... 19
Chapter 3 METHODOLOGY 3.1 Introduction………...…..………. 23
3.2 Research Methodology………...……….…...……… 23
3.3 Hardware……… 24
3.4 Python Programming and Python Integrated Development……...… 24
3.5 System Design Flow Chart……….…… 25
3.6 Flow Chart of Methodology………...…… 26
3.7 Object Detection using SSD Mobilenet v1………..…….. 27
3.8 Dataset Pre-processing………..……. 29
3.8.1 Collection of Dataset………...……... 29
3.8.2 Annotation………...……... 30
v
3.8.3 Binarization………...……. 31
3.9 Training………..…… 31
3.9.1 Compute Loss……….………... 31
3.10 Testing……….………. 32
3.10.1 Compute Precision and Evaluate Performance of Model Checkpoints………...………... 33
3.11 Export Retrained Model………...…… 34
3.12 Completion of Child Detection System……… 35
3.12.1 Types of Inputs………..…….. 35
3.12.2 Classifier………..…… 35
3.12.3 Triggering System………..…. 36
3.13 Experimental Evaluation of Child Detection System…………..… 36
3.13.1 Response Time………...….. 36
3.13.2 Maximum Distance of Detection………...….. 37
3.13.3 Precision………...…… 37
3.14 Project Management………...…….. 38
Chapter 4 RESULT AND DISCUSSION 4.1 Introduction………..…... 39
4.2 Object Detection on Jupyter Notebook……….. 39
4.3 Detection with SSD Mobilenet v1………...……... 40
4.4 Loss Graph………...………..……… 41
4.5 Model Evaluation………..………. 42
4.6 Experimental results ………...……...………...……. 46
4.6.1 Response Time………...…… 47
4.6.2 Maximum Distance of Detection………...…… 48
4.6.3 Precision………...….. 56
4.7 Discussion………..…… 61
4.8 Sources of Error………..… 61
Chapter 5 CONCLUSIONS AND RECOMMENDATIONS 5.1 Conclusions………...…………...……… 62
5.2 Recommendations………..…… 62
REFERENCES 63
APPENDIX A 68
APPENDIX B 72
APPENDIX C 73
APPENDIX D 75
APPENDIX E 78
APPENDIX F 82
APPENDIX G 84
APPENDIX H 85
vi
LIST OF TABLES
Table Page
Table 1.1 Comparison between the Advantages and Disadvantages of
Current Vehicular Occupant Detection Systems in the Market 4 Table 2.1 Difference between Machine Learning Tasks 13 Table 2.2 Speed and Accuracy of Pretrained Model 19
Table 3.1 Pretrained Model 28
Table 3.2 Distribution of Training and Testing Dataset 30
Table 3.3 Gantt Chart for FYP1 and FYP2 38
Table 4.1 Sample Ground Truth and Computed Data 42
Table 4.2 Result of Testing at Step 19608 46
Table 4.3 Response Time for 6 Candidates 47
Table 4.4 Child Detection with Varying Distances 49 Table 4.5 Adult Detection with Varying Distances 54 Table 4.6 Reproducibility Test of the Child Detection 58 Table 4.7 Reproducibility Test of the Adult Detection 58
Table 4.8 Terminology in Confusion Matrix 59
Table 4.9 Confusion Matrix of Child 59
Table 4.10 Confusion Matrix of Adult 59
Table 4.11 Theoretical and Experimental Precisions 60
vii
LIST OF FIGURES
Figure Page
Figure 1.1 Circumstances of Child Vehicular Heatstroke Death in
United States ( 1998 – 2016) 2
Figure 1.2 Sense-A-Life Vehicular Occupant Detection System 2 Figure 1.3 Hyundai Rear Occupant Alert System 3 Figure 2.1 Relationship between AI, ML and DL 7 Figure 2.2 Procedure for Building a Classification Model 9
Figure 2.3 Procedure in a Regression Task 10
Figure 2.4 Construction of Regression Line based on Samples 11 Figure 2.5 Characteristics of Machine Learning Models 13 Figure 2.6 The Processing Steps in a Perceptron 15
Figure 2.7 Architecture of ANN 16
Figure 2.8 Difference between Nueral Network and DNN 16 Figure 2.9 Hierarchy Feature Extraction in CNN 17
Figure 2.10 Architecture of RNN 18
Figure 3.1 Python IDLE Version 3.6 24
Figure 3.2 Logic Diagram of Child Detection System 25
Figure 3.3 Procedures in Methodology 26
Figure 3.4 Process Flow to Accomplish Object Detection Method 27
Figure 3.5(a) Sample Adult Images 30
Figure 3.5(b) Sample Child Images 30
Figure 3.6(a) Labelling Images 30
Figure 3.6(b) XML File 30
Figure 3.7 Command Window 31
Figure 3.8 Loss Values 32
Figure 3.9 Running Evaluation Command 33
Figure 3.10 The Computed Precision for Model Checkpoint at Step
19613 33
Figure 3.11 Child Detection System 35
viii
Figure 3.12 Experiment Setup Diagram 37
Figure 4.1 Results of Object Detection 39
Figure 4.2 Sample Image for Detection 40
Figure 4.3(a) Detection Outcome based on SSD_Mobilenet_v1 40 Figure 4.3(b) Detection Outcome based on Retrained model 40
Figure 4.4 Loss Graph during Training 41
Figure 4.5
Relationship between Predicted Bounding Box, 𝐵𝑝 and
Ground Truth Bounding Box, 𝐵𝑔𝑡 42
Figure 4.6 Performance of Model Checkpoints by Category (Adult) 44 Figure 4.7 Performance of Model Checkpoints by Category (Child) 44 Figure 4.8 Mean Average Precision (mAP) of Model Checkpoints 45
Figure 4.9 Setup of Experiment 47
Figure 4.10 Response Time for 6 Candidates 48
Figure 4.11 Results of Child Detection 57
Figure 4.12 Results of Adult Detection 57
ix
LIST OF ABBREVIATIONS
AI - Artifiial Intelligence ML - Machine Learning DL - Deep Learning
ANN - Artificial Neural Network DNN - Deep Neural Network
CNN - Convolutional Neural Network RNN - Recurrent Neural Network
IDLE - Integrated Development Learning Environment XML - Extensible Markup Language
HTML - Hyper Text Markup Language WAV - Waveform Audio
CSV - Comma Separated File
1
CHAPTER 1
INTRODUCTION
1.1 General Background
Heatstroke is defined as a situation whereby the body lost the ability to cool itself due to prolonged exposure to high temperatures (Mayo Clinic, 2017). The symptoms of heatstroke include high body temperature, normally achieved a temperature of 40oC or higher, nausea, vomiting, rapid breathing, racing heart rate, headache and so on. There are children dying from heatstroke each year after being left unattended in vehicles.
According to McLaren (2005), although under a relatively cool ambient temperature, the majority rise in temperature of a parked vehicle occurs at the first 15 to 30 minutes.
According to Kidsandcars (2017), the probability of children suffer from heatstroke in cars is much higher than that of adults. Core body temperature of children rises more rapidly under high temperature due to their greater body surface area to mass ratio as compared to adults. A child can get overheat at around 3 to 5 times faster than an adult.
Statistics show that there is a total of 700 child fatalities due to vehicular heatstroke since 1998 in the United States and 87% of them were aged under 3 years old. Figure 1.1 shows the circumstances of child vehicular heatstroke (Null, 2017).
2
Figure 1.1: Circumstances of Child Vehicular Heatstroke Death in the United States (1998 – 2016)
The issue of children dying of vehicular heatstroke has raised the attention of the public and lead to the invention of various types of devices to help remind drivers or caregivers. The typical product currently available in the market includes Sense-A-Life, a device that utilized pressure sensor to detect the presence of a child, and immediately alert driver that has left a child in the car through a mobile application and speakers. It is designed with simple installation and easy transfer between vehicles.
Figure 1.2: Sense-A-Life Vehicular Occupant Detection System (Adapted from Schlosser, 2016)
54%
28%
17% 1%
Circumstances
Forgotten Gained Access Left Intentionally Unknown
3
However, there are weaknesses in this invention. The usage of the pressure sensor cannot differentiate whether the force applied to it is a human or a load. This will cause the occurrence of false signal generation. Besides, it is designed to be installed in a baby car seat. It cannot detect the presence of a child who gained access to the car on their own.
Another invention is developed by Hyundai Motor named rear occupant alert system. The system implemented ultrasonic sensor to detect motion of child in the rear seats after driver leaves the vehicle and activate triggering system including horns sound, light flash and send text message to the driver. The system can work effectively even the child is not put in the baby seat (Newswire, 2017).
Figure 1.3: Hyundai Rear Occupant Alert System (Adapted from Muller, 2017) However, the system cannot generate the signal if the child is sleeping or not move.
In addition, the system will generate a signal to any motion regardless of whether the vehicular occupant is an adult or a child. This reduces the reliability of the system.
Current products in the market generally comprised of sensors to sense temperature and vehicular occupant detection as well as triggering system. Weaknesses of these products are the ability to detect correctly whether there is a vehicular occupant present in the car and whether he or she is a child or an adult. As such, a study should be done to develop a child detection system which can accurately detect the presence of a child in a car. Application of Artificial Intelligence (AI) technology in image recognition system is the reason for the idea of developing a child detection system to achieve the objectives of this research.
4
Table 1.1: Comparison between the Advantages and Disadvantages of Current Vehicular Occupant Detection Systems in the Market
Advantages Disadvantages
Sense-A-Life Ease to install Pressure sensor cannot differentiate between a child and a load
The region of detection is limited in the baby car seat only Hyundai Rear Occupant
Alert System
Detection is not limited to the baby car seat only
Cannot differentiate the occupant is an adult or a child
Fail to sense the occupant if there is no motion detected
1.2 Problem Statement
The problem encountered currently is the available sensors in the market cannot differentiate accurately the vehicle occupant is an adult or a child. The sensor can detect the presence of vehicular occupant and generate signal to alert car owner. However, the system cannot determine whether the occupant is a child or adult. This will increase the frequency of false signal generation.
The inaccurate sensor such as pressure sensor will activate triggering system to any load applied on it, generating the false signal to users, thus reducing the reliability of the detection system.
On the other hand, problem to be faced in this project is the qualities of input which will affect the accuracy of results. This problem may occur due to motion of children.
Other factor such as brightness, distance and angle from the camera to the target would affect the performance of the system as well.
5 1.3 Objectives
The objectives of this study are:
To provide a framework for the development of a child detection system with AI using object detection method.
To establish an image recognition system that can detect the presence of a child.
To develop a child detection system that is able to reduce the frequency of false signal generation.
1.4 Scope of research
This research focuses on developing an alternative way to detect vehicular occupant instead of using sensors available in the market. The system established will response to the presence of child only. This research will help to study the workability in using image recognition to differentiate a child against an adult. There are several challenges that may not be solved by using image recognition. Firstly, training an image recognition system requires a large amount of images as training data. Besides, training an image recognition model required the useage of Graphics Processing Unit (GPU) in order to speed up the process and obtain a more accurate retrained model.
6
CHAPTER 2
LITERATURE REVIEW
2.1 History of Artificial Intelligence
The field of Artificial Intelligence (AI) was coined by John McCarthy and two senior scientists: Claude Shannon and Nathan Rochester in 1956 at a conference, Dartmouth Conference, the first conference devoted to the subject (Buchanan, 2006).
They proposed that every aspect of learning or any other feature of intelligence can be precisely described, and a machine can be made to simulate it. According to Luger (2009), artificial intelligence (AI) is the science to enable the machine to accomplish the thing that requires intelligence. It promotes the use of a computer to do reasoning. AI can refer to anything from a computer programme playing a game to pattern recognition, image recognition as well as text and speech recognition by utilizing AI technology. The field of AI reached its major advance in the year of 1980, with the emergence of Machine Learning (ML), an approach to achieve AI (Dietterich & Michalski, 1983). Machine learning works on the principle of using algorithms to learn from examples. The machine is trained to learn from a large amount of data by itself, without explicitly programmed.
However, conventional machine learning techniques were limited in their ability to process natural data in their raw form (Luppescu & Romero, n.d.). The weakness had led to the blossom of Deep Learning (DL) in the 2000’s. DL is a set of methods that allow a machine to be fed with raw data and automatically discover the representations needed for detection and classification (Lecun, Bengio, & Hinton, 2015). DL architectures comprised of multiple processing layers to extract a useful representation of data. For example, in image processing, raw images are fed into the learning model. Initial layer extract edge detection, the second layer detects the motif, the third layer may assemble motifs into larger combinations that correspond to parts of familiar objects, and
7
subsequent layers would detect objects as combinations of these parts. In short, AI is any techniques that enable computers to replicate human intelligence. ML is the subset of AI involving the field of study that gives computers the ability to learn without explicitly programmed. Improvement of ML leads to DL, which allow a model to learn the representations of data with multiple levels of abstraction. Figure 2.1 shows the relationship between AI, ML, and DL.
Figure 2.1: Relationship between AI, ML, and DL
2.2 Machine Learning
According to Negnevitsky (2002), machine learning consists of adaptive mechanisms that enable the computer to learn from experiences or data exposed to it. The knowledge is improved by keeping making adjustment and correction on the error signal generated. The most famous machine learning mechanism include artificial neural networks and genetic algorithms. Machine learning is a field of study that focuses on computer systems that can learn from data. These systems are called models. Models can learn to perform a specific task by analysing lots of examples for a particular problem.
There are three modes to conduct machine learning, including supervised learning unsupervised learning and reinforcement learning.
2.2.1 Supervised Learning
A machine learning process whereby the model is to predict the provided output or target variables. In other words, all the target variables are labelled. Classification and regression are examples of the supervised learning problem, which the output variables are either categorical or numeric. Under supervised learning, machine learning can further be divided into classification task and regression task.
AI - Human intelligence exhibited by machines ML - An approach to achieve AI
DL - A technique for implementing ML
