advanced lane detection based on car driver assistance systems

By

Alexander Agung 11501057

BACHELOR’S DEGREE in

MECHANICAL ENGINEERING – MECHATRONICS CONCENTRATION FACULTY OF ENGINEERING AND INFORMATION TECHNOLOGY

SWISS GERMAN UNIVERSITY The Prominence Tower

Jalan Jalur Sutera Barat No. 15, Alam Sutera Tangerang, Banten 15143 - Indonesia

July 2019

Revision after Thesis Defense on 18 July 2019

Alexander Agung STATEMENT BY THE AUTHOR

I hereby declare that this submission is my own work and to the best of my knowledge, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma at any educational institution, except where due acknowledgement is made in the thesis.

Alexander Agung

_____________________________________________

Student Date

Approved by:

Dr. Rusman Rusyadi B.Eng, M.sc

_____________________________________________

Thesis Advisor Date

Dr. Maulahikmah Galinium S.Kom, M.Sc.

_____________________________________________

Dean Date

Alexander Agung

ADVANCED LANE DETECTION BASED ON CAR DRIVER ASSISTANCE SYSTEMS USING OPENCV WITH IMPLEMENTATION OF VEHICLE AND PEDESTRIAN

DETECTION

By

Alexander Agung

Dr. Rusman Rusyadi, B.Eng., M.Sc., Advisor

SWISS GERMAN UNIVERSITY

The goal of this thesis project is to be able to provide a reliable safety driver assistance system for drivers by detection of road lane boundaries where the source of the vision is from a camera which is mounted inside the car in order to get the vision since the system is a vision based system, the object detection should be able to classify whether it is a vehicle or a pedestrian with Deep learning method where the data is trained using TensorFlow. The lane detection and object detection system were constructed in a C++ language with Qt as the framework and the integration of both systems is using multithread. Deep learning is applied based on the machine learning approach it is selected as the method used in this research. By implementing lane detection and object detection, it enables the detection of position of the lane boundaries and the surrounding objects in the environment would be detected. Classification of object and lane detection system was tested using a data video recorded in Indonesia and developed using a dashcam/webcam as the vision.

Keywords: Lane Detection, Object Detection, TensorFlow, Deep Learning, Classification, Vision

System

Alexander Agung

© Copyright 2019 by Alexander Agung

All rights reserved

Alexander Agung DEDICATION

I dedicate this thesis to God My family

My friends

and for the future of this country I loved: Indonesia

Alexander Agung ACKNOWLEDGEMENTS

I am grateful to Almighty God for His grace and blessing throughout the entire process of this thesis work. I also would like to thank my parents and family members for giving me endless support, attention and financial needs during the process of doing this thesis work.

I would like to express my sincere thanks to Dr. Rusman Rusyadi B.Eng., M. Sc., as my Thesis Advisor which has been really patience, giving valuable advices and endless encouragement he showed me, I am indebted to him for his expertise and for all the valuable lessons that he taught me during the process of this thesis work

I would like to express my thanks to my fellow projects, Fransiska, Reyner Reynaldi Indarto, Luke Indracahya Setyawan and Chosua Glen for helping me. I would like also to thank my best friend Davin Edison and Richard Christo who motivates me and guide me during the development of the software. I would also like to thank all my colleagues in Basecamp and the other Mechatronics Batch 2015 for the help and support.

Without all those listed above, this thesis would not have been completed.

Alexander Agung

ABSTRACT ...3

DEDICATION ...5

ACKNOWLEDGEMENTS ...6

CHAPTER 1 - INTRODUCTION ... 13

1.1 Background ... 13

1.2 Thesis Problem ... 14

1.3 Thesis Objectives ... 14

1.4 Significance of Study ... 15

1.5 Thesis Limitations ... 15

1.6 Thesis Scope ... 15

1.7 Thesis Outline ... 16

CHAPTER 2 – LITERATURE REVIEW ... 17

2.1 Theoretical Perspectives ... 17

2.2 Lane Departure Warning System (LDWS) or Lane detection system ... 17

2.2 Machine Learning and Deep Learning ... 23

2.3 Vehicle and Pedestrian Detection with deep learning ... 23

2.4 Comparison for types of deep learning methods. ... 25

2.5 Neural Networks with types of pre-trained models ... 26

2.6 Multithreading ... 29

2.6.1 Thread vs Process... 29

2.6.2 Advantages and Disadvantages of Multithreading ... 30

2.6.3 Synchronization in multithreading ... 30

CHAPTER 3 - RESEARCH METHODS ... 32

3.1 Introduction ... 32

Alexander Agung

3.3 Research Framework ... 33

3.4 System design ... 34

3.5 Material and equipment ... 35

3.6 Software ... 36

3.7 Lane detection packages ... 37

3.7.1 System structure of lane detection process ... 38

3.8 Training configuration ... 45

3.9 Nvidia driver ... 45

3.10 Mathematical equation... 46

3.10.1 Perspective transformation calculation ... 46

3.10.2 Curve fitting calculation ... 47

3.10.3 Bounding box calculation ... 48

3.10.4 Softmax function ... 49

3.11 Data collection and analysis ... 50

3.12 Application and performance test ... 50

CHAPTER 4 – RESULTS AND DISCUSSIONS ... 51

4.1 Introduction ... 51

4.2 Live Testing Setup ... 51

4.3 Lane Detection System Test results ... 53

4.3.1 Daylight lane detection system testing ... 53

4.3.2 Night time detection system testing ... 60

4.4 TensorFlow ... 62

4.4.1 TensorFlow installation ... 62

4.4.2 TensorFlow build from source ... 64

4.4.3 Training procedure and results ... 65

Alexander Agung

4.5.1 Daylight object detection test results... 70

4.5.2 Night time object detection test results ... 75

CHAPTER 5 – CONCLUSIONS AND RECOMMENDATIONS ... 78

5.1 CONCLUSIONS ... 78

5.2 RECOMMENDATIONS ... 79

GLOSSARY ... 80

REFERENCES ... 81

APPENDIX A – LANE DETECTION TEST RESULT ... 83

APPENDIX B – OBJECT DETECTION TEST RESULT ... 87

APPENDIX C – TRAINING TEST RESULT ... 90

Curriculum Vitae... 91