5-2: Pre-post-trial group comparison and pre-post task comparison on reaction time 49 5-3: Mean tonic SCL, frequency of phasic SCRs and HR of caregivers in IJA trials 51. A-5: Mean tonic -SCL, frequency of phasic SCRs and HR of caregivers in RTN and IJA trials,.
Autism Spectrum Disorder and Intervention
Intelligent Intervention Systems for Young Children with ASD
Socially Assistive Robotics (SAR) for Children with ASD
- Literature Review on SAR for Children with ASD
- Multi-session Studies with SAR Systems
- Prediction of performance in HHI based on gaze distribution in HRI
Their results showed that the NAO could arouse the social interest of children with fashion disorders and thus potentially be used for intervention. This study was conducted with 6 children with ADD in a school setting, where these children experienced up to ten 40-minute non-directive sessions with experimenters once a week.
Computer System for Children with ASD
- Literature Review on Computer-Mediated Human-Human Interaction for children with ASD
- Computer-Mediated Caregiver-Child Interaction
Many children with disruptive disorders are attracted to computer and robotic systems that can be designed to provide unified intervention and objective, quantitative performance tracking. Several researchers, including the authors, have designed several autonomous systems to train RJA skills for young children with ADHD.
Anxiety Prediction for Children with ASD based on Physiological signal
- Affective computing
- Affective computing for autism
- Affective computing based on fNIRS
People with ASD often have difficulty determining the emotional states of themselves and others [92]-. There was work on predicting the affective status of people with ASD based on physiological signals [90], such as EDA, PPG, ECG, EMG, EEG, etc.
Analytical Modelling of Affective Data
Labeling the affective status of individuals with ASD has been a challenging problem, partly due to the challenge of obtaining reliable self-reports of affective status from them [102]. Some studies used self-reports such as Self-assessment Manikin (SAM) [129] scales as labels [127], and other works used standard emotion elicitation databases such as IAPS and GAPED [102] as labels, or used unsupervised learning . techniques [103].
Research Objectives
- Specific Aim 1: Exploration on the design of multi-session robot-mediated joint attention intervention
- Specific Aim 2: Predicting Social Performance in Real Life from Gaze Distribution in HRI
- Specific Aim 3: Design of a Computer-Mediated Caregiver-Child Interaction System
- Specific Aim 4: Prediction of stress for individuals with ASD based on fNIRS signal
There have been works on predicting the affective status of people with ASD based on physiological signal [90], such as EDA, PPG, ECG, EMG, EEG, etc. So far, no study has been reported on predicting the stress status of individuals with ASD based on the fNIRS signal, which we propose as future work to fill the research gap.
Multi-session Study with Waitlist Control Group Design with a SAR
- System Design
- Humanoid Robot
- Target Monitors
- Gaze tracking sub-system
- Supervisory Controller
- Measures
- Procedures
- Study Timeline
- Session Format
- Planned Data Analyses
- Participants
- Results
- Time × group interaction analyses
- Pre-post intervention comparison of all participants
- Examination of participants who did and did not improve
- How changes in STAT performance related to baseline clinical characteristics variables
- How changes in within-system performance related to clinical characteristics
- Discussion
Crowell, “Clinical Use of Robots for Individuals with Autism Spectrum Disorders: A Critical Review,” Res. Wilson, “Prevalence of anxiety and mood problems among children with autism and Asperger syndrome,” Autism, vol.
Exploration of Design of Multi-Session Robot-Mediated Joint Attention Intervention for
NORRIS OVERVIEW
- System Description
- Supervisory controller
- Robot administrator
- Gaze tracking subsystem
- Target monitors
- Intervention task design
- Performance measurements
In a single-target trial, the robot asked the participant to look at the left or right target monitor. A trial was successful if the participant looked at target(s) within a response time window by following the robot's cues (target hit). Depending on the participant's performance, trial duration ranged from 10 seconds (the participant hit the target immediately after the first prompt) to 1 minute (all prompts were required).
Multi-session Study I
- Participants Characteristics
- Study timeline and protocol
- Results
-2) A higher hit rate means more successful trials and a lower prompt level means trials were completed with fewer prompts. To compare changes in hit rate and prompt level across robot-mediated intervention and natural maturation ( Figs. 3–3 ), we calculated the differences between IPG-A1 and IPG-A2 and the differences between WCG-A0 and WCG-A1. The decrease in hit rate and the increase in prompt level from WCG-A0 to WCG-A1 may reflect decreased RJA skills during the waiting period, which may be due to the reduced novelty effect and thus a reduced engagement with NORRIS.
Analyses and reflection on Study I
- System Capture Ratio (SCR)
- Analysis of session-SCR in Study I
- Analysis of trial-SCR of sessions in Study I
- Summary of findings from Study I
Below, we describe the session and trial SCR analyzes in Study I, followed by a description of Study II. To better understand participants' commitment to a better solution, we examined the mean trial-SCR trend in Study I. NUMBERS OF SINGLE-OBJECTIVE PARTS AND DOUBLE-BS IN S3 AND S4 COMPLETED BY PARTICIPANTS IN STUDY I.
Multi-session Study II (Study II)
- Intervention timeline
- Performance-based adaptive intervention session
- Participant Characteristics
- RJA performance analysis
As such, we hypothesized that a more varied trial design would help maintain participant engagement within a session. For stroke rate (higher is better), there was a significant improvement in the time factor, with a large effect size. For the fast level (lower is better), there was a significant decrease in the time factor, with a large effect size.
SCR in Study II
- Validation of SCR in Study II
- Analysis of session-SCR in Study II
- Analysis of trial-SCR in Study II
There was no significant difference between two trial SCRs across all trials in Study II (Figs. 3–8). In Study I, trial SCR began to decrease significantly after Trial 4 (Figs. 3-5 and Tables 3-7). Trial-SCR trends in Study II (solid black: all trials; green: trials with S-D-T; red: trials with S-D-D; blue: trials with S-S-D).
Discussion and Conclusion
There are four RJA tasks in an ESCS test where an image (i.e. the target) is placed on both the left and right sides of the participant. Robins, “The Effectiveness of Using a Robotics Classroom to Promote Collaboration Among Groups of Children with Autism in an Exploratory Study,” Pers. Berryman, “The impact of stress and anxiety on people with autism and developmental disorders,” in Behavioral issues in autism, Springer, 1994, pp.
Predicting Response to Joint Attention Performance in Human-Human Interaction Based on
Problem Background
- System and Task Design
- Data Collection
Four web cameras, Cam 1 to Cam 4, constituted the gaze tracking module, surrounding the center of the participant's chair with a radius of 90 cm. A NAO robot stood in front of the participant's chair, performing deictic gestures to initiate the joint attention task. The environment of the system was symmetrical about the line connecting the center of the participant's chair and the NAO robot.
Features Generation
- Gaussian Mixture Model (GMM) of Head pose
- Hard Histogram of Head Pose
- Soft Histogram of Head Pose
- Features of Head Pose Motion
Therefore, a three-component GMM was adopted to model the distribution of main pose vectors ([𝑦𝑎𝑤, 𝑝𝑖𝑡𝑐ℎ, 𝑟𝑜𝑙𝑙]) in an HRI session. Therefore, fitting the head pose vectors from an HRI session into the three-component GMM leads to a 30-dimensional feature vector. The way GMM models the distribution of head pose in an HRI session is meaningful but crude.
Machine Learning Framework
- Label Thresholding
- Feature Pre-processing
- Feature Fusion
- Evaluation methods and metrics
Then the membership vectors calculated for all head pose vectors in an HRI session were added and normalized to obtain a soft histogram for this session, as shown in Equation 3-2. 𝛥𝐻𝑛,𝑖= 𝐻𝑛,𝑖+1− 𝐻𝑛,𝑖 Equation 4-3 We used the above-mentioned hard histogram and soft histogram to model the distribution of head postural movement in each HRI session. Soft histogram of head pose + soft histogram of movement of head pose 4.3.4 Evaluation methods and metrics.
Results
- Features of Head Pose
- Features of Head Pose Motion
- Feature Fusion
- Cross-subjects validation results
The hard histogram was better than soft histogram when modeling the distribution of head position. The optimal K for hard and soft histogram of head position motion were both 48, overall. Both hard and soft histogram Table 4-3 Metric mean (std. dev) of hard histogram of head position.
Conclusion
Although several technology-based JA studies have shown potential, they primarily focus on the response to joint attention (RJA). Another important component of JA, the initiation of joint attention (IJA), has received less attention from a technology-based intervention perspective. In this work, we present an in-depth computer-mediated caregiver-child interaction (C3I) system that helps children with ADHD practice IJA skills.
System Development
- C3I Design Steps
- Overview of System Architecture
- Peripheral Modules
- Central Controller
- Monitor & Speaker Array
- Gaze Tracking Subsystem
- Body Tracker
- Caregiver Tablet App
- Interaction Protocol
Then, in the distraction condition, a muted distraction video was played on a randomly selected monitor to distract the child from the caregiver. Finally, the caregiver had to confirm the child's response to trigger the reward video on monitor 1. The child was expected to look towards the caregiver to find out why the video was paused.
Experiments
- Study Design
- Recruitment
- Measurements
- Response Time and Distraction Duration
- Caregiver Stress
- User Experience Questionnaire
It was the time between when a caregiver paused the video and when the caregiver acknowledged the child's response (see Figure 5-7). In other words, each time a bouncing ball was activated, the response time was penalized approximately 3 seconds. Distraction duration was a covariate because it could also influence the child's response time.
Results and Discussion
- Quantitative Metrics
- Responsive Time and Distraction Duration
- Stress of Caregivers
- Quantitative Metrics-User Experience Questionnaire
At the end of the second set of activities, do you think your child looked at you faster when the video was paused. 5-9 Frequency of phasic-SCRs variation of caregivers in IJA trials (Mean value with 10% SD as bar). 5-10 Heart rate variation of caregivers in IJA trials (Mean value with 10% SD as error bar) Baseline.
Conclusion & Future Work
- Conclusion
- Limitation
- Future work
Zheng et al., “Design of an Autonomous Social Orientation Training System (ASOTS) for Toddlers with Autism,” IEEE Trans. Kim et al., “Social robots as integrated reinforcers of social behavior in children with autism,” J. Warren et al., “Brief report: development of a robotic intervention platform for young children with ASD,” J.
Classifying stress of individuals with ASD based on functional Near-Infrared Spectroscopy
Problem description
120] to work on for the following reasons: First, it had standardized timing of stimuli and recording of raw light intensities, including both low and high wavelengths. Second, the labeling is clear, with no need for further clarification or additional conditions, such as the work of Hudak et al. The starting position of the sliding window spanned from the start to the end of the task period.
Proposed models
- Neural networks
- XGBoost
The proposed 1D Inception-Resnet was expected to perform better than 1D Inception because of the remaining links during it. Inception, Resnet, and Inception-Resnet blocks can be duplicated and pooled into their respective models to balance variance and bias. To avoid overfitting, we reduced the plateau learning rate during the training period by a factor of 0.5, a tolerance of 10, to a minimum of one tenth of the initial learning rate.
Evaluation
It has been suggested that more convolutional layers with smaller kernel sizes are easier to train and perform better, compared to less convolutional layers with larger kernel sizes. For all three models, the final classification layer had the same number of neurons as the number of output classes. There are generally two types of feature selection methods: filter (model independent, less computing power) and wrapper (select features based on interaction with a classifier.
Results
The generated feature dimensionality was more than 900. Thus, the feature selection was applied to eliminate features that were not relevant to output labels. After careful hyperparameter tuning, the predictive performance using HbO, HbR, and Hb was shown in Tables 6-1, 6-2, and 6-3, respectively. We can see that one-dimensional Inception-Resnet achieved the best performance with HbO, HbR, and Hb averaging 64.6%, respectively.
Discussion and Conclusion
Third, this work is only the first step in designing a stress classification for individuals with ASD. Due to the uncertainty brought about by COVID-19, participants with ASD were difficult to bring in for data collection. When appropriate, participants with ASD should be recruited to create a dataset of fNIRS based on which the proposed models will be validated before it can be adopted by any autonomous systems designed for people with ASD.
Major Contributions
Technological Contributions
To the best of our knowledge, this is the first work that attempts to quantitatively bridge the gap between HRI performance and HHI performance for children with ASD. Next, as none of the existing works could address social skills training for children with ASD as well as alleviating caregiver anxiety in parallel, we developed a new Computer-Mediated Caregiver-Child Interaction (C3I) system to fill this gap. Finally, since no study has been reported on predicting stress of individuals with ASD based on fNIRS signal, we proposed predictive models to fill this research gap.
Societal Contributions
Dautenhahn, “Engaging: Assisted play with a Kaspar humanoid robot for children with severe autism,” in International Conference on Telecommunications, 2018, p. robotics, 2009, p. .
In memory of critical moments in Ph.D. career
