Ailbot: a Respiratory-Focused Symptom Checker Chatbot For Children
Gabrielle Mae V. Arco, Ken Ivan T. Cheng, Pamela S. Chong, and Chico Andre G. Olaguer
De La Salle University Integrated School
*corresponding author: [email protected] Judith J. Azcarraga
De La Salle University-Manila
Abstract: Inadequate access to healthcare, accompanied by the spread of COVID-19 has significantly affected most Filipinos. The prolonged quarantine period in the Philippines due to this virus restricts the movement and lifestyle of people. Thus, it makes medical appointments difficult to arrange. The ever-growing number of healthcare chatbots can provide access to healthcare check-ups quickly. These systems are still in their infancy stages, with the majority focusing on the mental and emotional well-being of people and catering to the general public. This study aims to develop a chatbot geared toward the respiratory health of Filipino children. Through the use of Chatfuel, Ailbot, a symptom checker chatbot was developed and integrated into Facebook. Ailbot aims to determine the likelihood of its user having common respiratory ailments, such as Common Colds, Bronchial Asthma, Pneumonia, Influenza (Flu), and Acute Sinusitis. To evaluate the Ailbot’s usability, functionality, and humanity, five Filipino children aged 8 to 12 were asked to converse with the bot and share some feedback about their conversational experience. Results indicated that Ailbot is excellent in its Usability metric, attaining an average System Usability Scale (SUS) score of 82.5. Consequently, Ailbot proved to be functional and very humane based on a self-made questionnaire. Using thematic analysis, four common points of feedback were present through all the interviews – ease of usage, issue of being a non-digital native, adult assistance, and medical knowledge. Additionally, feedback from the parents and medical experts was noted for future reference to aid the improvement of Ailbot.
Keywords:Chatbot, Healthcare, Symptom Checker, Healthcare Chatbot, Telehealth
1. INTRODUCTION
Chatbots are systems that use natural language user interfaces to mimic the interactions of humans. These systems use machine learning to learn from previous activities to improve their response accuracy and recognize different replies from users (Pereira and Diaz, 2019).
Chatbots may be integrated into various platforms. In each platform, they serve a specific purpose when interacting with their users. They act as a virtual conversational agent that
may provide different kinds of services such as healthcare provisions. Chatbots developed specifically for healthcare are called healthcare chatbots. There are three common archetypes for these – chatbots for diagnosis, chatbots for prevention, and chatbots for therapy (Jovanovic, 2019) – many of which allow for more personalized content that provides users with responses tailored to their profile. All of these make chatbots powerful tools that can be incorporated into telehealth.
Telehealth is the use of technology to provide healthcare services from long distances by connecting patients to doctors through video conferences, live monitoring of patients’ health, etc. It is a technology that improves healthcare services and provides access to health provisions (Gajarawala & Pelkowski, 2021). It bypasses certain obstacles such as distance issues and resource expenditure (gasoline, time, etc), and allows patients to reach medical professionals much faster and easier.
In the Philippines, people experience inequity in healthcare provision and services (Banaag et al., 2019). The people living in poverty face a challenge, usually requiring much more healthcare needs but receiving less service.
Collado (2019) suggests that medical attention or comprehensive checkups may be hard to access for some people due to their proximity to medical facilities. Telehealth services may alleviate such problems. There are many free-to-use healthcare chatbots online, such as Ada health, Babylon health, and more which are all symptom checker chatbots. However, an issue stands that these chatbots are mainly catered for adults’ use. This implies that the chatbots might not be children-friendly or easy to navigate for children. This is one of the gaps in the Telehealth sector that must be addressed so that children may also have access to healthcare while they are at home.
This study presents a symptom checker chatbot catered to Filipino children. The symptom checker may help identify the possible respiratory ailments its user might have.
Ailbot asks a series of questions on whether or not a certain symptom is experienced by the child. After which, the system displays the possible respiratory ailments the child may have.
2. METHODOLOGY
The development and analysis of the chatbot are divided into six sections: Chatbot Medical Database, Chatbot Design, Chatbot Development, Expert Evaluation, Chatbot Testing, and Data Analysis.
2.1. Chatbot Medical Database
Ailbot’s medical database was selected from medical sources such as WebMD Health (WebMD, n.d.) and Mayo Clinic (Mayo Clinic, 2021).
2.2. Chatbot Design
Ailbot’s conversational flow is patterned on other chatbots similar to BookBuddy by Ruan et al. (2019)
Figure 1 displays Ailbot’s conversational flow, which has four phases: Introduction Phase, Check-up Phase, Diagnosis Phase, and Conclusion Phase. The Introduction Phase asks the user and their parent for consent to collect sensitive information, such as name and age, and medical information like allergies, vitamins, and medication. After which, the Ailbot will ask if the user is feeling well and will ask whether to proceed to the Check-up Phase. The Check-up Phase has three parts: General Wellness questions, Symptom questions, and Medical Attention questions. General Wellness questions ask for current vitamins and medication, weight, allergies, and pain area. The Symptom questions part which expects a “yes” or “no” answer follow the approach of an expert system. An expert system can be primarily defined as
“a computer system that emulates the decision-making ability of a human expert, which aims to solve complex problems by reasoning knowledge” (Tan, 2017). Then, the Medical Attention questions are only asked if the user possesses all symptoms of a certain ailment. These questions determine if they need to immediately see a doctor. If a user needs medical attention, the chatbot will ask the user to see a doctor immediately. In the Diagnosis Phase, it provides a receipt that shows the likelihood of its user having Common Colds, Bronchial Asthma, Pneumonia, Influenza, and Acute Sinusitis. Lastly, in the Conclusion Phase, the chatbot asks the user to rate their experience and concludes the chat.
Figure 1
Ailbot’s Conversational Flow
Additionally, pictures depicting the symptoms were created and integrated into Ailbot. This was done so that Ailbot could better appeal to children and help them understand the symptoms more.
2.3. Chatbot Development
Ailbot was developed using Chatfuel, a natural language understanding platform. It was deployed on Facebook and Messenger under the page named “Ailbot.”
Shown in Figure 2 is a sample conversation with Ailbot on Messenger.
Figure 2
Ailbot in Messenger
2.4. Expert Evaluation
To validate the accuracy of Ailbot’s medical database, consultations were organized with three medical professionals: a medical officer, a general practitioner, and a pediatric pulmonologist. The medical professionals evaluated and shared their feedback on the materials used in this study.
These materials include the list of symptoms under the five respiratory ailments commonly experienced by Filipino children and Ailbot’s conversational flow. Ailbot itself was also tested, but only by the pediatric pulmonologist. This series of verifications serves to provide credibility and corrections to the chatbot’s framework and database.
2.5. Chatbot Testing
The chatbot testing phase was conducted to assess Ailbot’s ability to converse with its users and to determine its user’s likely respiratory ailment. A Zoom meeting was conducted with each of the five children participants aged 8-12 years old who were accompanied by their parents. The rationale behind the age range is that the children should be literate and capable of communicating with a chatbot. Only five children were interviewed as this phase is somewhat similar to an “initial testing.”
Participants were asked to converse with the chatbot while pretending to be sick. Afterward, participants were asked the interview questions. During this time, a powerpoint presentation was used to display the questions. Lastly, the parents and children were asked to give additional comments/feedback that they might have. The entire meeting was recorded with the consent of the participants and their parents.
2.6. Data Analysis
A mixed analysis was done to assess the feedback given by the participants during the interviews. The quantitative data would come from the ratings of participants on the SUS questionnaire self-made functionality and humanity questionnaire. The results from the SUS questionnaire were scored and interpreted according to Sauro
as the 50th percentile in the scores. These scores are tabulated with the corresponding grade and adjectival rating. Both of these values are other forms of representing the scoring. The grade was made by Sauro and Lewis similar to academic grades (Sauro, 2018). Meanwhile, the adjectival rating was made by Bangor et al. by associating 1000 SUS scores with a 7-point adjective scale, which was then reduced to a 6-point adjective scale. Both of these score representations differ in their criteria score range. Consequently, the results from the functionality and humanity questionnaire were summated and the mean was determined for both metrics to compute the percentage score. Lastly, the average scores and percentages for the SUS and self-made questionnaire were computed and tabulated as shown in Tables 1 and 2.
The qualitative data came from the participant’s suggestions, feedback, and reactions/expressions when they used the chatbot. Thematic analysis was used to categorize similar responses to assess the chatbot’s overall performance.
3. RESULTS AND DISCUSSION
Presented below are the preliminary results with five children participants ages 8-12 years old. These results were used to study the chatbot’s overall usability, functionality, and humanity. Additionally, several features that were liked and disliked by the children were identified.
3.1. SUS Questionnaire
The System Usability Scale (SUS) was designed to assess the usability of systems (e.g. computer hardware, television, mobile devices, etc.) (Sauro, 2011). This reliable form of evaluation has been used by a multitude of studies because of its effectiveness in assessing the usability of chatbots (Abd-Alrazaq, 2020). For the same reason, the SUS questionnaire was used to assess Ailbot’s usability
The questions from the SUS questionnaire were modified to be easily understood by children. Additionally, the questionnaire made use of the 5-point Likert scale compliance with the SUS’ method of scoring.
Table 1
System Usability Scale Scores from Filipino Children ages 8-12
Participant # Score Grade Adjectival Rating
1 82.5 A Excellent
2 77.5 B+ Good
3 92.5 A+ Excellent
4 85 A+ Excellent
5 75 B Good
Average 82.5 A Excellent
Shown in Table 1 are the SUS scores that were calculated based on articles by Sauro in 2011 and 2018.
Additionally, The scores range from 75 to 92.5, with an average of 82.5 and a standard deviation of ≈ 6.85. On average, the participants gave an A grade and an excellent rating on the usability of Ailbot.
3.2. Functionality and Humanity Questionnaire
The results shown in Table 2 were determined from the participant’s scoring on the 6-question Functionality and Humanity Questionnaire. Questions 1 to 4 were for functionality, while questions 5 and 6 were for humanity. The questionnaire made use of the 5-point Likert scale.
Table 2
Functionality and Humanity Scores from 5 Filipino Children ages 8-12
Participant # Functionality Score Humanity Score
1 85.00% 100.00%
2 80.00% 100.00%
3 90.00% 100.00%
4 75.00% 100.00%
5 80.00% 80.00%
Average 82.00% 96.00%
The average percentage was determined for both metrics. The scores are 82.00% for functionality and 96.00%
for humanity. The functionality scores range 75.00% to 90.00%. Meanwhile, the humanity scores ranged from 80.00% to 100.00%. The standard deviation for functionality is 5.70% and 8.94% for humanity.
3.3. Quantitative Analysis System Usability Scale
The range of SUS scores (75-92.5) indicates that the chatbot’s usability performed better than the 50th percentile score (68). The standard deviation (6.85) infers that the scores provided by each of the participants are close to the average SUS score. Furthermore, the average SUS score of the participants (82.5) is approximately rated at 90th-95th percentile rank of the scale which is an acceptable score for its usability. These values indicate that Ailbot is very much usable by Filipino Children ages 8-12.
Functionality and Humanity Scale
Based on the range and average of the Functionality (Range: 75%-90%; Average: 82%) and Humanity (Range:
80%-100%; Average 96%) scores, it is inferred that Ailbot is deemed functional and very humane by the respondents.
Supportively, the low standard deviation calculated from the Functionality and Humanity metrics infers that the mean of both metrics are reliable and numerically near to the participants' individual scores. In summary, Ailbot is usable, functional, and humane, based on user feedback.
3.4. Thematic Analysis Ease of usage
All of the participants described their experience using Ailbot as easy and not confusing in any way.
Translation (TL): “It’s easy to talk to. I understand it.” – participant 2
Issue of being a Non-digital Native
2 of the 5 children suggest that children their age may not be able to use it easily. This is because there are children who do not have access to devices or Facebook, which could affect their ability to learn using Ailbot.
“Some of my classmates aren’t ‘techy’. Mostly they don't have FB(Facebook)also.”– participant 4.
The rest of the participants answered that Ailbot could be learned by children of their age easily because of the current home learning setup, which requires the majority of students to learn how to navigate devices.
“At my age, you can answer those questions already and everything is online.”– participant 3
Adult Assistance
The majority of the participants sought help from their parents in the General Wellness questions by Ailbot, which asks the children whether they take medications/vitamins and to specify what, and their weight in kilograms. The children can be seen calling their parents for help in answering these two items. After this phase, the participants responded to the line of questioning in the symptom checking phase with ease.
TL: “I think that’s where he called for help. Maybe it would be better if there are choices, or to separate the medications from vitamins.”– parent of participant 4
“I disagree(needing help from a parent) because I know how it(Ailbot)works.”– participant 3
Medical knowledge
Most of the participants suggested that Ailbot’s knowledge on the ailments are enough to assess the symptoms. However, they admitted that they don’t know much about the terms used during the whole conversation with the chatbot.
“I think the chatbot’s knowledge is enough but I’m not really familiar with the medical terms.”– participant 3
Parents’ Feedback
Some of the parents who accompanied the participants during the interview gave feedback and suggestions as well. A common suggestion from these parents was to add a statement that gives the child advice on what to do with the chatbot’s assessment.
TL: “Like you could say ‘you may or may now see a doctor’..
parang depending on how many days you’ve had colds ‘all you need is rest’... Then after a certain amount of days you should see a doctor .”– parent of participant 3
In addition, they also suggested adding an option to choose a language [English / Filipino] at the start before it surveys the children of their health.
TL: “Add an option for a preferred language of Filipino/
English at the start for wider reach.”– parent of participant 4 Other than these recommendations, the parents found the chatbot very helpful and fun for the children. It was helpful because Ailbot is able to provide questions tailored to ask whether the child is feeling well or what exactly the child is currently going through in an empathetic and easy-to-understand manner.
TL: “I think it's very useful. I like it because as a parent it’s a bit hard to ask step by step the symptoms of the kid. With this, it’s like the questions are already prepared for you. When it comes to children, they can’t easily target or explain what they’re feeling, but with this (Ailbot), it’s like the questions are already itemized so that the child can just choose and that’s why I like it.”– parent of participant 1
Positive reactions were observed among the parents and children. It was because parents noticed that the children kept using it even though the conversation was over with the chatbot. This could be observed during the interview where the children interacted with the chatbot a lot and laughed along with their parents.
“I had fun because I think he or she(referring to Ailbot)is nice!”– participant 1
“Let’s go back to the zoom already… Let’s stop and go back to the zoom meeting… He doesn’t want to stop using it.”– parent of participant 1
3.5. Medical Professionals Comments and Suggestions
Recommendations and suggestions from three medical professionals – a school physician, an adult cardiologist, and a pediatric pulmonologist – were used in the development of Ailbot. This was done to improve Ailbot and ensure the accuracy and correctness of its medical database.
Online consultations were conducted via Zoom and communication was coursed through Gmail.
The design and scope of the chatbot’s list of illnesses were suggested by a school physician. This limits the scope and focuses on only five respiratory ailments. Some medical resources (e.g. Merck Index) and concepts, such as SOAP (Subjective, Objective, Assessment, and Plan) which may improve Ailbot’s medical database were also introduced.
Aside from these, it was also mentioned to add visuals (photographs and animations) and use a platform different from Facebook and Messenger due to privacy issues associated with them. Lastly, minor comments were made on the conversational flow. For the Introduction Phase, it was recommended to reiterate consent before starting the Check-up Phase. In the next phase, it was suggested to use differential diagnosis, pain scales, and questions about allergies, medications, vitamins, and medical history.
Another physician, who is also an adult cardiologist, suggested that the aim of Ailbot should only be to aid patients, but not to help them self-diagnose and self-medicate. A set of symptoms and triggers for the respiratory ailments was also provided to improve Ailbot’s medical database. Finally, it was recommended to seek advice from a pulmonologist.
Another advice from a pediatric pulmonologist is to further improve Ailbot’s medical database by basing it on more reliable sources, such as medical journals and research articles. Some of these were provided by the pulmonologist.
These include works from the Global Initiative for Asthma (2021), Philippine Academy of Pediatric Pulmonologists, Inc.
and Pediatric Infectious Disease Society of the Philippines, Inc. (2021), Pappas (2020), and Wald (2021).
4. CONCLUSIONS
Limited access to healthcare has been a problem in recent times with COVID-19. The objective of this study is to find a way to alleviate this through the creation of a symptom checker chatbot for Filipino children. However, the chatbot, Ailbot, was not meant to be a replacement for
doctors, but a supplement in its role as a symptom checker.
Ailbot was created in Chatfuel and its conversational flow consists of four phases: Introduction Phase, Check-up Phase, Diagnosis Phase, and Conclusion Phase. The chatbot focuses on five common pulmonary diseases among Filipino children (ages 8-12), which are Common Colds, Bronchial Asthma, Pneumonia, Influenza, and Acute Sinusitis. Tested on five children respondents, the preliminary results show some positive responses from the children and their parents. With a SUS’ Excellent average score of 85, Functionality Score of 82%, and Humanity Score of 96%, it can be concluded that Ailbot is already capable of communicating with children. However, according to medical professionals, much can still be improved with the chatbot’s knowledge of the five ailments, and how questions are asked. Lastly, preliminary results and feedback from the respondents and medical professionals show that there is a need to further improve the chatbot’s conversational flow and database.
6. REFERENCES
Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., and Denecke, K. (2020). Technical Metrics Used to Evaluate Health Care Chatbots: Scoping
Review. Retrieved from
https://www.jmir.org/2020/6/e18301
Banaag, M. S., Dayrit, M. M., & Mendoza, R. U. (2019).
Health inequity in the Philippines.New Frontiers in Regional Science: Asian Perspectives, 163–174.
https://doi.org/10.1007/978-981-13-6268-2_8 Jovanovic, M., Baez, M., & Casati, F. (2020). Chatbots as
conversational healthcare services. IEEE Internet Computing, 1–11.
https://doi.org/10.1109/mic.2020.3037151 Gajarawala, S. N., & Pelkowski, J. N. (2021). Telehealth
benefits and barriers.The Journal for Nurse Practitioners,17(2), 218–221.
https://doi.org/10.1016/j.nurpra.2020.09.013 Global Initiative for Asthma. (2021).2021 Global Strategy
for Asthma Management and Prevention.
https://ginasthma.org/gina-reports/.
Mayo Foundation for Medical Education and Research.
(2021). Mayo Clinic.https://www.mayoclinic.org/
Pappas, D., MD, JD. (2020, April 6).The common cold in children: Clinical features and diagnosis.
UpToDate. Retrieved April 2, 2022, from
https://www.uptodate.com/contents/the-common-co ld-in-children-clinical-features-and-diagnosis.
Pereira, J., & Díaz, Ó. (2019). Using health chatbots for behavior change: A mapping study. Journal of Medical Systems, 43(5).
https://doi.org/10.1007/s10916-019-1237-1 Philippine Academy of Pediatric Pulmonologists, Inc. &
Pediatric Infectious Disease Society of the Philippines, Inc. (2021). 2021 Clinical Practice Guidelines in the Evaluation and Management of Pediatric Community-Acquired Pneumonia.
https://papp.org.ph/pcap-resources/.
Ruan, S., Willis, A., Xu, Q., Davis, G. M., Jiang, L., Brunskill, E., & Landay, J. A. (2019). BookBuddy:
turning digital materials into interactive foreign language lessons through a voice chatbot.
Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale,pp. 1-4, June 24-25 2019, New York, NY, USA. Association for Computing Machinery.
https://doi.org/10.1145/3330430.3333643.
Sauro, J., PhD. (2011, February 3).Measuring Usability with the System Usability Scale (SUS). Measuring U.
Retrieved April 2, 2022, from
https://measuringu.com/sus/.
Sauro, J., PhD. (2018, September 19).5 Ways to Interpret a SUS Score. Measuring U. Retrieved April 2, 2022, fromhttps://measuringu.com/interpret-sus-score/.
Tan, H. (2017, September). A brief history and technical review of the expert system research.IOP Conference Series: Materials Science and Engineering, 012111.
https://doi.org/10.1088/1757-899x/242/1/012111 Wald, E., MD. (2021, April 7).Acute bacterial rhinosinusitis
in children: Clinical features and diagnosis.
UpToDate. Retrieved April 2, 2022, from
https://www.uptodate.com/contents/acute-bacterial- WebMD - Better information. Better health. (n.d.). WebMD.
Retrieved April 2, 2022, from https://www.webmd.com/