SectionVI.2discussed our inferences on the scaffold statistics, i.e., the number of adaptive conversational scaffolds received by students with different behavioral profiles, as reported in Table VI.5. This section discussed our plans about redesigning or improving certain trigger conditions in future research, to account for between-group differences and to ensure even more meaningful and task-relevant adaptive scaffolding.
SectionVI.3reported on the responsiveness and strategic usage of scaffolds by tracking student and group behaviors before and after scaffolding. The case studies in SectionVI.4provide further clues on how to refine certain scaffolds to benefit students in future design iterations.
The general similarity observed in inter-scaffold intervals between the four groups, as observed in Sec- tionVI.1.2, is an improvement on previous scaffold design iterations (Munshi et al.,2022b,a), and may be attributed to more uniform triggering parameters and the inclusion of inter-scaffold interval as a separate pa-
rameter within the current design framework, which ensured that students did not receive their next scaffold unless a minimum set amount of time had passed since their previous scaffolding.
The results from the case studies (SectionVI.4) also provide opportunities for improving scaffold trig- gering factors like the priority assignment algorithm, for instance, with respect to Scaffold 2: Level-1 and Scaffold 9 (see discussion in SectionVI.4). The temporal analysis in SectionVI.3, beyond helping us un- derstand the between-group differences in the impact of scaffolding, also provided insights into temporal changes in such impact, for instance, the case of C2 after receiving Scaffold 5 multiple times. Such results present further scope to introducefadingfor certain types of scaffolds based on student responses, to main- tain engagement with the task and future responsiveness to scaffolding. We also found that the Read→Build scaffolds (Scaffolds 1, 2 and 3) and the Quiz→Build Correct Link Annotation Feedback (Scaffold 5) were generally followed by high response rates, with certain groups exhibiting more effective cognitive strategies and metacognitive monitoring behaviors in theafterintervals, eventually linking the strategic use of scaffolds to improvements in their causal models.
We now summarize some of our future plans on revising the current adaptive scaffolding framework based on the findings from this chapter. Scaffold 1was used by C1 as a corrective hint. C2 showed low response rates to this feedback, instead checking quiz results and being unsuccessful at using Quiz→Build to fix their shortcut links. C3 used the scaffold strategically and effectively, with the case study even showing very high affective valence alongside the strategy use for this group. But interestingly, C4 did not receive this scaffold despite the presence of shortcut links at certain points on their maps. In future design iterations, we may use out understanding of a student’s current behavioral profile, as discussed above, to provide more feedback that is tailored to the characteristics of such behavioral groups. For instance, if a student is identified to be an inefficient information generator (C2) and does not respond to Scaffold 1, they may receive additional scaffolding that guides them through the Read→DebugShortcutLink process. If a student is detected as an experimenter (C4), then their priority assignment algorithm may be modified dynamically (as discussed in SectionVI.2) if it is observed that they have shortcut links on their map but are not receiving any Shortcut Link Feedback due to the higher priority assigned to Scaffold 3.Scaffold 2was more effective for the disengaged group, who used the feedback strategically and also showed a decrease in disengagement after scaffolding.
So, the trigger condition for Scaffold 2 may be placed at a higher priority order for a student who is identified as disengaged in a future study. Tinkering behavior after scaffolding may also be monitored, so that students can receive additional feedback (e.g., we noted that C4 students changed the sign of incorrect links instead of deleting them in the current results). Scaffold 3 was useful for both C3 and C4 groups to develop a more coherent Read→Build strategy. We discussed how this feedback may also have accounted for the high potential usage durations in the experimenter group. In SectionVI.2, we discussed how the trigger condition
forScaffold 4may be improved to include an additional component which tracks incorrect link annotations already present on the student’s map prior to scaffolding. This is based on the finding for C4, who used the scaffold to correctly perform new link annotations but did not review previous incorrectly annotated links.
Otherwise, this scaffold was successful in helping C2 developquiz→link annotationas a strategy to debug and improve their maps. ForScaffold 5, which was received a large number of times by C1, we noted higher disengagement after successive rounds of scaffolding. Therefore, this scaffold may be reduced to a one-time feedback, or faded with time if disengagement is diagnosed, or may even be redesigned to provide additional feedback on how to combine the two types of link annotations (Scaffolds 5 and 6) to isolate potentially incorrect links. Scaffold 6generally showed low response rates but the case study also suggested that this feedback may be made more meaningful to the student if Mr Davis first asks them to take another quiz, before going into the Quiz→Build coherence behavior, thereby ensuring that Build actions performed since their last quiz is also reflected in the new quiz results, further giving them a better understanding of the current state of their map and form a plan to debug errors in a more coherent manner from the quiz results. The trigger condition forScaffold 7may be assigned a higher priority when students show inefficient information generator characteristics, with a follow-up Read→Build scaffold after the Quiz→Read that then helps the student to extract the correct causal relations from the text they are reading. ForScaffold 8andScaffold 9, an important component of our future plans is to perform a better evaluation of the affective components of this feedback by further validating the detector predictions using ground truth labels, as discussed above.
Scaffold 8 had low response rates while Scaffold 9 had high response rates in the current study. The case study suggested that the system should monitor students’ use of Scaffold 9 in a more fine-grained manner after the feedback is delivered, to identify the type of situation observed in the case study (viz., the consecutive deletion of multiple links without obtaining evidence from the Science Book or Quiz Results). As discussed in SectionVI.4, such situations should trigger additionaldiagnosticfollowed bystrategicfeedback to teach students how to develop a more evidence-driven solution construction process.
CHAPTER VII
Conclusions and Future Work
This dissertation contributes to the field in two primary directions: (1) Design & development, and (2) Re- search. We discuss the specific contributions along these two directions in some more detail below.