Our overarching research question for scaffold evaluation was:RQWhat is the impact of adaptive scaffolds triggered by the framework on learners’ self-regulated learning (SRL) processes?
We started the data analysis towards answering this question by first defining our learning outcome metrics in Betty’s Brain (SectionV.2.1) and conducting an exploratory data analysis (SectionV.2.2) by studying students’ overall learning outcomes, behavioral and affect indicators from the classroom intervention. The data-driven insights obtained from this exploratory analysis allowed us to formulate a more targeted research question (SectionV.2.3) and a corresponding set of targeted analyses for scaffold evaluation (SectionV.2.4).
V.2.1 Learning Outcome Metrics
We developed two metrics, one summative and one formative, to measure students’ learning outcomes from the Betty’s Brain intervention study.
1. Summative Assesment: Normalized Pre-to-post test learning gains, calculated as Post score−Pre score Max score−Pre score: This measure, derived from grading students’ pre- and post-test responses using a pre-defined rubric, helps us to evaluate how well the intervention helps students learn their science content;
2. Formative: Map scores, calculated as# number of correct minus incorrect causal links in a student’s causal map at any point in time during the intervention. This measure helps us track the correctness of students’ causal models built in Betty’s Brain over time, and provides us with a more direct measure of their causal modeling abilities in the system. We compute thefinal map scoresachieved by each student at the end of the intervention as a second summative measure of their overall performance during the intervention.
V.2.2 Exploratory Data Analysis
An exploration into students’ overall learning outcomes from the Betty’s Brain intervention was performed using the metrics outlined in SectionV.2.1. The findings, reported in SectionVI.1, revealed that the study par- ticipants (n=55) as a whole did not exhibit significant pre-to-post learning gains. However, large variances in both pre-to-post gains and final map scores were observed, which prompted us to apply an unsupervised learning approach to determine if students clustered into groups based on their behavioral differences and if this explained the wide variations in performance. In addition, behavior differences could also help us determine if the adaptive feedback provided were differentially used by the different groups. The results, reported in SectionVI.1, revealed four clusters: Cluster 1(n=6) consisting of disengaged learners,Clus- ter 2 (n=19) consisting of inefficient information appliers,Cluster 3(n=22) consisting of strategizers, andCluster 4(n=8) consisting of experimenters or tinkerers who exhibited trial-and-error behaviors while working on Betty’s Brain. We discuss the clustering approach and the labeling of each of these clusters in the next chapter.
V.2.3 Research Questions for Scaffold Evaluation
The clusters observed from the exploratory data analysis were used to frame a more targeted research question to study and compare the impact of adaptive scaffolds from our scaffolding framework on the four groups (clusters) of students.
RQ:How did students from the four different clusters respond to receiving the different types of adaptive scaffolds (listed in SectionIV.4.4?More specifically,
(a)Were students in each clusterresponsiveto the scaffold? In other words, did they appear to follow the actionable recommendation provided in the scaffold by then performing the activities suggested by the mentor?
(b)Additionally, did the subsequent behaviors of students in each cluster convey astrategic useof the scaffolds? For instance, if the objective of a scaffold was to teach a specific cognitive-metacognitive regulation strategy, did the students show a change in their relevant model-building activities, behaviors and performance after they received the scaffold? If the scaffold also included an affective component, was it also possible to detect a change in learner emotions?
V.2.4 Temporal Analysis for Scaffold Evaluation
To answer the above research questions, we performed temporal analysis for student clusters (findings re- ported in SectionVI.3) that tracked the change in their use of suggested strategies, their causal modeling performance, (and emotions, as applicable) as they received conversational feedback in the form of adaptive
scaffolds from the agents. For this purpose, we created sequences ofscaffold-triggered’before’ and ’after’
intervalsacross a student’s learning timeline in Betty’s Brain, so that we could compare student behaviors before receiving scaffolds to their behaviors after receiving scaffolds.
Here, theafter-interval for an adaptive scaffold started when the scaffold was given to the student and continued up to the time the student received their next scaffold from the system. Similarly, thebefore- intervalfor an adaptive scaffold considered the time starting from when the last (previous) scaffold was given to the time when they received the current scaffold (Munshi et al.,2022b). For example, consider a student who received two adaptive scaffolds during the course of their learning session - Scaffold 1 at timetiand Scaffold 2 at timetj. For Scaffold 1, the student’sbeforeinterval would be[0,ti]andafterinterval will be [ti,tj], where the time 0 represents the start of the current session. Similarly, for the second delivery of a Scaffold 2, thebeforeinterval would be[ti,tj]andafterinterval will be[tj,end], whereendrepresents the end time of the session.
To determine theresponsivenessandstrategic useof the adaptive scaffolds received by students in each cluster, we studied the change in their cognitive activities (as relevant to the triggering context and conversa- tion content of the delivered scaffold), their map scores (performance), and the likelihood of their prevailing affect states (when relevant and accurately available) in thebeforeandafterintervals for each feedback they received. The specific activities and strategy use to be assessed depended on the type of cognitive or metacog- nitive strategy (and as relevant, affect state) that was supported by the agent’s feedback. For example, in the case of the Read→Build Incorrect Link scaffold, we first checked whether the learner followed the mentor agent’s suggestion in the scaffold by reading the suggested page(s). Then we also assessed strategic usage by checking if their causal modeling performance improved in the following interval, suggesting that they were able to use the strategic reading to develop more effective model construction behaviors that fixed prior errors in their causal model. We also studied the impact of scaffolds after the first, second, ...,nthtime it was received by students in a cluster.
Chapter VI presents the findings from data analysis conducted following the methodology outlined in this chapter. The results from an exploratory analysis of students’ learning outcomes are reported in Sec- tionVI.1.1, followed by a study of the affect indicators available from the study data, especially focusing on data reliability and its implications on further analysis. This is followed by feature construction for cluster analysis using causal modeling behaviors in Betty’s Brain (SectionVI.1.3). The clustering procedure and its results are reported in SectionVI.1.4, which shows four behavioral profiles or groups among students. Then SectionVI.2presents the statistics on adaptive scaffolds received by these groups, and discusses the implica- tions of these numbers on future refinement of the scaffold design framework. SectionVI.3goes deeper into the temporal analysis of the impact of scaffolds received by students in each group, with discussions on group
and student responsiveness / strategic use of each scaffold. SectionVI.4presents case studies exploring the behavioral evolution and scaffold use of two students from the classroom study. Finally, while SectionsVI.3, VI.3, andVI.3discuss our inferences on the effectiveness of scaffolds and the implications on future scaffold designs in Betty’s Brain, SectionVI.4discusses some of the major future research directions that can extend the findings from this dissertation, while also outlining the limitations from the current evaluation study that can be explored further in future work.
CHAPTER VI
Results and Discussion
Exploratory analysis of the data collected from the classroom study was conducted to investigate students’
overall learning outcomes, study their behavior and affect indicators, and then study the impact of adaptive scaffolds provided in the system.