In-depth log analysis was also performed by the researcher with the collected conversation logs during the end-user testing. This aims to recognize the times VHope’s sub-modules and its features failed, how they failed, and what are the possible fixes for them.
6.7.1 Welcome Message
The welcome messages of VHope contains more than one sentence. The general idea for this is to provide and show as much support as it can so that the user will feel more listened to and will then be able to share anything. Also, these messages are pre-defined and designed to be covering broad scope. They are pre-defined as using the generative model for it loses its purpose of asking the user since the model cannot specifically compose a question when asked to. These messages are also covering broad scope as to not limit he user on what to share and give freedom to them to share any story or topic they feel sharing at the moment.
1 SP -1014
2
3 VHope : Hi ! I am willing to listen to any of your stories today . Please go on .
4 User : Hello
5 VHope : I don ’ t know what to say to that !
6 User : You are s u p p o s e d to say hi back
Listing 6.8: VHope Model-S not understanding the response and can’t respond properly to a simple greeting
1 HP -1024
2
3 VHope : Hello ! What would you like to share about your day ? You can tell me a n y t h i n g .
4 User : Hello
5 VHope : Thanks for c h a t t i n g with me I hope to see you again tomorrow , bye !
6 User : hi
7 VHope : What do you want to talk about ?
8 User : A n y t h i n g will do
Listing 6.9: VHope Model-H not understanding the response and can’t respond properly to a simple greeting
But having too many sentences to answer, users were confused and were either able to answer accordingly by starting their story or just responds to the greeting of ’Hi’ or ’Hello’. Also, VHope seems not aware of what common responses to common phrases such as ’Hello’, ’Thank you’, and ’Goodbye’ should be. This can be solved by training the generative model on common phrases and its acceptable responses. That way, the model will learn how to respond even if the users got confused and was only able to answer back with a short greeting.
6.7.2 Abot name
The name ’Abot’ keeps on appearing in some conversations. Sometimes VHope introduces itself as Abot, and sometimes pertains to the user it is talking to as Abot. This further confuses the user as it appears out of nowhere and the name
’Abot’ is not really common.
1 SP -1001
2
3 User : Let ’ s talk about s o m e t h i n g
4 VHope : That ’ s a good idea . I will look into that .
5 User : We just i n t e r v i e w e d someone for a course a s s i g n m e n t
6 VHope : I see , well , I guess I ’ ll start by i n t r o d u c i n g myself I ’ m Abot, a chatbot d e s i g n e d to help you improve your overall well - being , but on the side I also enjoy reading stories and
w a t c h i n g a lot of movies !
7 User : What is your f a v o r i t e movie ?
Listing 6.10: VHope Model-S introducing itself as Abot
1 SP -1012
2
3 User : Okay ... it was nice c h a t t i n g with you too .
4 VHope : What is your f a v o r i t e hobby ?
5 User : I like playing video games .
6 VHope : Nice to meet you Abot! What would you like me to call you ?
7 User : You can call me xxx .
Listing 6.11: VHope Model-S referring to its user as Abot
Recorded logs from the conversations of a well-being chatbot named ’Abot’
(Sia et al., 2021) were included in the Well-being Conversations dataset. From these conversations, Abot always start by introducing itself to the user and the user even calls it Abot in his/her responses. During model training, removing the term Abot while cleaning the conversation lines was skipped. Because of this, the model learned the term Abot as it appears almost on all the conversations.
6.7.3 Repetitive Responses
A noticeable response that keeps repeating in the conversations is the output ’I see. What else can you say about that?’ which appears almost at the end of conversations. This is not a newly generated response from the generative model
in the retrieval-based model and thus can’t provide the necessary input for the generative model. This ERROR catching feature is added so that the chatbot won’t crash every time this ERROR appears.
1 SP -1012
2
3 User : Oh , I hope you feel better !
4 VHope : I ’ m sorry to hear that . What h a p p e n e d ?
5 User : Nothing h a p p e n e d to me .
6 VHope : I see . What else can you say about that ?
7 User : I have nothing to say to you anymore
8 VHope : I see . What else can you say about that ?
9 User : I am sleepy but it ’ s too early to sleep
10 VHope : I see . What else can you say about that ?
11 User : I want to sleep but I have to do school work
Listing 6.12: VHope Model-S generating repetitive responses due to an ERROR in the retrieval model
Users responded to this repetitive responses both negatively and positively.
Some of them got irritated and ended the conversation right away. Most of the users that reacted this way are those bored and doesn’t enjoy talking with VHope as they always responds with short inputs. On the other hand, there are users that kept on sharing their stories and thoughts even while this repetitive response persists. According to their evaluation, they are already enjoying chatting with VHope so they were able to continue talking with it and just thinks of the response as showing VHope’s interest in listening to them. These users are those attached to the ideal objective of VHope - which is to actively listen to anything while empathizing with them as a form of mental support.
6.7.4 Continuous Sharing
1 HP -1001
2
3 VHope : Alright ! Just one last q u e s t i o n before we go , did you find our c o n v e r s a t i o n helpful ?
4 User : Somehow yes
5 VHope : I don ’ t know if I can handle that
6 User : Okay , how are you ?
7 VHope : That ’ s alright ! The parable begins with a l e g e n d a r y sword crafter known all over the world because of the e x t r a o r d i n a r y swords that he created
Listing 6.13: VHope Model-H continuously sharing a parable to its user
Data is the most important component of a machine learning model. The generative model was trained with Well-being conversations collected from actual conversation of a retrieval-based chatbot and its users. Thus, the pattern and words learned by the generative model are greatly influenced by how this initial chatbot is modeled to respond to its users.
Learned from Abot, VHope sometimes just continuously shares a whole story or movie in a single input. Some users find it amusing but others were also confused as it were unrelated to the topic at hand. Additionally, Abot’s users kept on using the ’I’ word in their stories which may have affected VHope’sfirst person responses. Again, this can be resolved if a better more objective-appropriate conversational data is used.
6.7.5 Single-turn conversations
The base of the generative model used for VHope performed really well on single- turn dialogues. But this time, it is trained and used for multi-turn conversations.
The role of the retrieval-based model is to guide this generative model to not lose the topic at hand.
VHope’s performance is somewhat successful as more than half its generated responses seem relevant to the users’ inputs according to the experts’ evaluation.
But it is really noticeable that VHope kept changing the topic which again confuses its users. This may have resulted from VHope’s ability to only focus on the current turn of lines, not remembering other important words or topics discussed from the first part of the conversation. For this, a possible solution is to keep a collection of words for the generative model’s use.
6.7.6 Forgetting past offers
1 HP -1014
2
3 VHope : I ’ ll give you some tips to improve your mood !
4 User : okay
5 VHope : I think y o u ll like it here !
6 USER : Where
Listing 6.14: VHope Model-H offering tips but forgetting to state what it is and the user actually asking where the tip is
VHope was able to generate responses that offers help or tips, may it be related to well-being or not. It also asks if the user wants to hear some story. But related to the finding that VHope only understands the current turn, it seems to forget what it has offered and responds with another unrelated output. This irritates the users and they try to ask about it again during the conversation but VHope is not capable of doing so. It keeps on generating new and fresh responses.
1 HP -1016
2
3 User : Because i am l e a r n i n g
4 VHope : Nice ! I ’ ll give you some tips to help you improve !
5 User : ok
6 VHope : I ’ m not sure if you are serious . Do you have any hobbies ?
Listing 6.15: VHope Model-H offering tips but forgetting to state what it is
6.7.7 Ending the conversations
There are many instances that VHope tried to end the conversation in the middle.
Some users try to ask why and wants to stick around to talk more while others openly accept it and just end the conversation. It is still unclear what triggers VHope’s goodbye statements but the retrieval-based model used for VHope has a keyword for ending their stories - the end. Unfortunately, this command was not shared to the users during their testing so a proper ending of conversations were not accommodated. Conversations just ended when users stopped talking or when the error, resulting to repetitive responses, persist. One theory of this abrupt ending of conversations is due to the training data of Well-being conversa- tions thanking the user for chatting not just at the exact end of the conversations.
There are still other sentences after it. Thus, model learned that the ’Thank you for chatting with me’ statement is not just for the end of the conversation.
1 HP -1018
2
3 VHope : You seem to be s t r u g g l i n g . Is that right ?
4 User : yes
5 VHope : Why do you think you are s t r u g g l i n g ?
6 User : i ’ m s t r e s s e d and i don ’ t know what to p r i o r i t i z e
7 VHope : Thanks for c h a t t i n g with me I hope to see you again tomorrow , bye !
Listing 6.16: VHope Model-H offering tips but forgetting to state what it is and the user actually asking where the tip is
1 HP -1024
2
3 User : studies
4 VHope : I don ’ t know what that is .
5 User : Am I on the right course ?
6 VHope : Thanks for c h a t t i n g with me It was a p l e a s u r e meeting you Let ’ s talk again tomorrow , bye !
7 User : hello
Listing 6.17: VHope Model-H offering tips but forgetting to state what it is and the user staying to still chat with it