development of web-based augmentative and alternative

(1)

By

Timothy Aditya Sutantyo 11502012

BACHELOR’S DEGREE in

Information Technology

Engineering and Information Technology

SWISS GERMAN UNIVERSITY The Prominence Tower

Jalan Jalur Sutera Barat No. 15, Alam Sutera Tangerang, Banten 15143 - Indonesia

July 2019

Revision after the Thesis Defense on July 17^th2019

(2)

STATEMENT BY THE AUTHOR

I hereby declare that this submission is my own work and to the best of my knowledge, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma at any educational institution, except where due acknowledgement is made in the thesis.

Timothy Aditya Sutantyo

_____________________________________________

Student Date

Approved by:

Dr. Maulahikmah Galinium, S.Kom., M.Sc.

_____________________________________________

Thesis Advisor Date

Dr. Eka Budiarto, S.T., M.Sc.

_____________________________________________

Thesis Co-Advisor Date

_____________________________________________

(3)

DEVELOPMENT OF WEB-BASED AUGMENTATIVE AND ALTERNATIVE COMMUNICATION (AAC) TOOLS FOR PEOPLE WITH RETT SYNDROME IN

ACTIVITIES OF DAILY LIVING (ADL)

By

Timothy Aditya Sutantyo

Dr. Eka Budiarto, S.T., M.Sc.

SWISS GERMAN UNIVERSITY

Rett Syndrome occurs between one in 10000 to 15000 girls in Indonesia. In Hasan Sadikin Hospital, there is a girl that suffers from Rett Syndrome. Since Rett Syndrome patients have little-to-no speech capabilities, as well as hand movements, it is currently difficult to communicate with Rett Syndrome patients, especially with regards to Activities of Daily Living (ADL). Hence, a method to communicate with Rett Syndrome patient is being proposed, so that it would be easier for patients to communicate with the caregiver using Augmentative and Alternative Communication (AAC). Two of the methods are gaze-tracking and using head-pose. The methods tested consists of head-pose algorithm, gaze-tracking with Dlib, gaze-tracking with Haar-cascades, and EyeTribe, each tested under different scenarios first before tested to the patient. EyeTribe, head-pose and gaze-tracking algorithm with Dlib Library achieves good results, with the best results being at 50cm distance and with high light intensity, however the Rett Syndrome patient still needs training in order to use the application.

Keywords: EyeTribe, Gaze Tracking, Head Pose, Rett Syndrome, ADL, AAC

(4)

(5)

DEDICATION

I dedicate this thesis to God, my family, lecturers who have supported me, friends, and also Rumah Sakit Hasan Sadikin.

(6)

ACKNOWLEDGEMENTS

I wish to thank my parents for giving birth to me, and hence allowing me to learn a lot in this university, as well as instilling life skills to me to be able to survive. I also want to thank my classmates during my study in SGU, even though they can be blunt and hard to me, since this means an opportunity to learn, whether with regards through my academics, also with my life skills. I would also thank you to Mr. Maula and Mr. Eka, who have been very patient with me during my thesis work, as well as giving me guidances which allow this thesis to be completed. Finally, I also want to thank my favorite virtual character from Final Fantasy Series, Celes Chere, who have

“supported” me during my studies.

Finally, my thesis would not be complete without the assistance of Jesus Christ. He has done everything to make sure I manage to do this thesis on time, and reminding me to complete the thesis when I am stuck in completing the thesis.

(7)

STATEMENT BY THE AUTHOR... 2

ABSTRACT...3

DEDICATION... 5

ACKNOWLEDGEMENTS... 6

TABLE OF CONTENTS...7

LIST OF FIGURES...11

LIST OF TABLES... 13

LIST OF EQUATIONS... 15

CHAPTER 1 - INTRODUCTION... 16

1.1. Background... 16

1.2. Research Problem...18

1.3. Research Objective...18

1.4. Significance of Study... 19

1.5. Research Questions... 19

1.6. Hypothesis...19

1.7. Scope... 19

1.8. Limitation...20

2 CHAPTER 2 - LITERATURE REVIEW... 21

2.1 Rumah Sakit Hasan Sadikin...21

2.2 Augmentative and Alternative Comunication (AAC)...21

2.3 Rett Syndrome...22

2.4 Eye Gaze Communication...24

(8)

2.5.2 Webcam...26

2.6 Head Pose Algorithm... 26

2.7 Prototyping Method...26

2.8 System Design and Life Cycle (SDLC)... 27

2.9 Related Works...27

2.9.1 BrainControl AAC... 27

2.9.2 Accent 1000 + NuEye Tracker...27

2.9.3 I-12... 28

2.9.4 Communicator 5 + PCEye Tracker... 28

2.9.5 TAP and TAPC... 30

2.9.6 Viability of Eye Tracking to support communication to patients with motor impairments... 30

2.9.7 Applications of Eye-tracking to different-abled individuals...30

2.9.8 Price comparison between existing products... 31

CHAPTER 3 – RESEARCH METHODS... 32

3.1 Architecture Diagram...32

3.2 Activity Diagram...33

3.3 Use Case Diagram...34

3.4 ERD Diagram...46

3.5 Functionality testing...46

3.6 User Acceptance Test (UAT)...50

3.7 Research Overview... 54

3.8 Algorithms...55

3.8.1 Using Dlib Facial landmark (Algorithm 1)...55

3.8.1.1 Modification to the algorithm... 57

(9)

3.10 Results Gathering for the algorithm...71

3.11 Gathering results from the observation at Hasan Sadikin Hospital... 73

3.12 Interview with the caregivers... 74

CHAPTER 4 – RESULTS AND DISCUSSIONS...76

4.1 First Visit to the hospital... 76

4.2 Experiments with Normal People... 76

4.2.1 Results of EyeTribe...77

4.2.2 Results of Gaze-tracking algorithm using Dlib (Algorithm 1)... 78

4.2.3 Result of Gaze-tracking with Haar-like features (Algorithm 2)... 80

4.2.4 Results of Head Pose (Algorithm 3)... 81

4.3 Discussions of the result with normal people...82

4.4 Second Visit to the hospital...87

4.4.1 Comments from the doctors... 87

4.4.2 Updated Website... 87

4.4.3 Observations with normal people in the hospital...90

4.4.4 Observations and results with diffable patients...91

4.4.5 Observations and results with Rett Syndrome Patient... 92

4.4.6 Results comparison between normal people, diffable people, and Rett Syndrome Patient... 94

4.4.7 Functionality Testing Result... 95

4.4.8 User Acceptance Test Results...95

4.4.9 Results of Interview with the caregivers...96

CHAPTER 5 – CONCLUSIONS AND RECCOMENDATIONS... 98

5.1 Conclusions... 98

5.2 Recommendations... 98

(10)

CURRICULUM VITAE... 104