MIXED-METHODS IN BEHAVIORAL RESEARCH:
ONLINE LEARNING AND HUMAN-COMPUTER INTERACTION
Harry Budi Santoso, PhD
Faculty of Computer Science, Universitas Indonesia
Depok, 19 November 2015
http://edgardoreyescalderon.blogspot.com
Professional Contribution
Chief Editor of Jurnal Sistem Informasi (JSI)
Program Chair of ICACSIS 2014 & 2015
Program Chair of Developing Online Education Seminar & Workshop 2014
Organizing Committee of the 1st and 2nd International Conference on Human- Computer Interaction and User Experience
Reviewer of Asia Pacific CHI UX 2015 (@ Symposium)
Reviewer of American Society for Engineering Education Annual Conference
Reviewer of ASEE/IEEE Frontiers in Education Annual Conference
Reviewer of ICACSIS Annual Conference
Reviewer of ICAICTA
Invited as a reviewer @ Journal of Educators Online
Invited as a reviewer @ International Review of Research in Open and Distance Learning
Research Interests
Online Learning/Computer Assisted-Instruction
Human-Computer Interaction/User Experience
Metacognition/Self-Regulated Learning
Engineering/Computer Science Education
Journal Writing Experience
Computers in the Schools (resubmitted, 2015)
The International Review of Research in Open and Distance Learning (published, 2015)
Journal of Pre-College Engineering Education Research (published, 2014)
Journal of Educators Online (published, 2014; accepted, 2015)
MERLOT Journal of Online Learning and Teaching (published, 2014)
International Education Studies (published, 2013)
International Journal of Engineering Education (published, 2013)
Design and Technology Education: An International Journal (published, 2013)
Journal of STEM Education: Innovations and Research (published, 2013)
Journal of Educational Technology & Society (published, 2012)
International Conferences
The 23rd International Conference on Computers in Education 2015, Hangzhou, China
The 3rd International Conference on User Science and Engineering 2014, Shah Alam, Malaysia
The 2013 ASEE/IEEE Frontiers in Education conference, Oklahoma City, Oklahoma, USA.
The 2013 American Society of Engineering Education (ASEE) annual conference, Atlanta, Georgia, USA
The 2012 ASEE/IEEE Frontiers in Education annual conference, Seattle, Washington, USA
The 2011 IEEE/ASEE Frontiers in Education annual conference, Rapid City, South Dakota, USA
Why do we need to publish?
Why do we need to publish?
austinkleon.com
The steps to get our article published
Conducting Research
Submitting the Research Paper
Before Submitting to Journal Publication
After Submitting to Journal Publication
HIGH QUALITY RESEARCH TO
HIGH IMPACT JOURNAL PUBLICATION
Interdisciplinary Nature of Online Learning &
HCI Research
Computer Science
Education
Psychology
Sociology
Anthropology
Engineering
Online Learning Framework
The Content of Human-Computer Interaction
(SIGCHI.ORG)
Types of Behavioral Research
(Rosenthal & Rosnow, 2008)
Descriptive Investigations
Keyword: Describe a situation or condition Typical Methods: …
Relational Investigations
Keyword: Identify relations between … (two or more variables) Typical Methods: …
Experimental Investigations
Keyword: Identify causes of a situation or condition
Typical Methods: …
Defining Mixed Methods
“Involved integrating quantitative and qualitative approaches to generating new qualitative approaches to generating new
knowledge and can involve either concurrent or sequential use of these two classes of methods to follow a line of inquiry.” – Stange K et al (2006).
“Integrating quantitative and qualitative data collection and
analysis in a single study or a program of enquiry.” – Creswell et
al 2003.
What is Mixed-Method Research?
focusing on research questions that call for real-life contextual understandings, multi-level perspectives, and cultural
influences;
employing rigorous quantitative research and rigorous qualitative research;
utilizing multiple methods (e.g., intervention trials and in-depth interviews);
(National Institutes of Health, Office of Behavioral and Social Sciences Research)
Download:
http://isites.harvard.edu/fs/docs/icb.topic1334586.files/2003_Cres
well_A%20Framework%20for%20Design.pdf
RESEARCH EXAMPLE
Computer Self-Efficacy, Cognitive Actions, and Metacognitive Strategies of High School Students While Engaged in Interactive
Learning Modules
Harry B. Santoso
Background
Educators and policy makers are engaged in an effort to improve the teaching of STEM subjects in the United States.
Introductory concepts of computer science, for example programming, are difficult to learn (Denning, 2004) and unattractive/unappealing (Jepsen and Perl, 2002).
Research suggests that computer applications can be used to increase learning and keep learners interested.
Initiatives have been made to promote the use of ICT in education:
The NSF Cyberlearning: Transforming Education program has objective “to better understand how people learn with technology and how technology can be used productively to help people
learn…” (NSF, 2011, p. 1).
Rationales
Self-Regulated Learning (SRL) as a major construct in educational research learning process
Despite the growing interest in SRL research, few studies have investigated the degree to which the students are aware of their
thinking process while working on interactive learning module (ILM) in high school level.
A very limited instrument is available to investigate SRL skills while learning using ILM.
Learning using computer applications requires SRL skills higher than learning with instructor in classroom (e.g., Chang, 2005).
Suggestions are needed to improve Interactive Learning Modules
Computer-Self-Efficacy, Cognitive Actions, and Metacognitive Strategies in SRL Framework
Self-regulated learners are “metacognitively, motivationally, and behaviorally active participants in their own learning process” (Zimmerman, 1989).
“SRL is a complex, situated, dynamic process involving individuals learning in context” (Butler & Cartier, 2005)
Self-Regulated Learning in Context
Motivation Metacognition
Behavior
Computer
Self-Efficacy Planning
Strategies Monitoring Strategies
Regulating Strategies
Cognitive Actions
Research Questions
How is students’ computer self-efficacy (CSE) related to
cognitive and metacognitive strategies while using interactive learning modules (ILM)?
Sub-question: What is the relative importance of CSE with regards to its contribution toward students’ cognitive actions and
metacognitive strategies while using ILM?
How do students’ plan and monitor their cognitive actions, and regulate their monitoring strategies during learning with ILM?
Sub-question: How do high and low CSE students plan and monitor
their cognitive actions, and regulate their monitoring strategies
during learning with ILM?
The Study Participants and Context
School Selection:
Participant Selection:
100 students from both schools completed all activities in this study.
Three modules for each class were selected to be used by considering the relevance of the modules to this study.
School Class
Logan High School Programming 1A and Math 1 InTech Collegiate High School Physics
Features of the Modules
Features Boolean Logic Minimum Spanning Tree Modeling Using Graphs
Readings √ √ √
Instructions √ √ √
Exercises √ √ √
Level of difficulties √ √ √
The Use of Multimethods
This research will use three data collection methods:
Online survey instruments
Demographic questionnaire
Computer Self-Efficacy questionnaire
Self-Regulated Computer-Based Learning questionnaire
Interactive Learning Module screen captured videos
Interviews
Computer Self-Efficacy Questionnaire
The survey will be used to understand students’ judgment of capabilities to use computers in different situations (Compeau, Higgins, and Huff, 1999; Marakas, Yi, & Johnson, 1998)
This questionnaire was adapted from the work of Durndell, Haag, &
Laithwaite (2000).
The CSE questionnaire responses ranged from 1 to 5 (i.e., 1 = not at all true of me and 5 = very true of me).
Scale Cronbach’s Alpha
(Original)
Cronbach’s Alpha (The Study)
Beginning Skills .930 .866
Advanced Skills .880 .919
File and Software Skills .900 .813
Self-Regulated Computer-Based Learning Questionnaire
The survey was developed to capture students’ perception of their cognitive actions and metacognitive strategies while learning using interactive learning modules.
This questionnaire was adapted from the work of Lawanto (2011) based on Butler and Cartier’s SRL theoretical model (Butler & Cartier, 2005; Cartier & Butler, 2004).
Measurement scales of EDQ items ranged from 1 to 4 (i.e., 1 = almost never, 2 = sometimes, 3 = often, and 4 = almost always).
Scale Cronbach’s Alpha (The Study)
Planning Strategies .694
Cognitive Actions .812
Monitoring Strategies .878
Regulating Strategies .814
Data Collection Procedures
Completing Demographic
& CSE online surveys Completing SRCBL
online survey Learning with the modules
Recorded using screen-capture software
Selected students were interviewed
Data Analysis
Answering Research Question 1 and Sub-question
Research
question How is students’ computer self-efficacy (CSE) related to cognitive and metacognitive strategies while using ILM?
Procedure Method Purpose
The mean values of CSE and SRCBL items will be calculated using
descriptive statistics and graphical views
Profiling of CSE, cognitive, and metacognitive strategies.
Correlation tests will be conducted by using
Pearson tests
To measure the relationships between:
(1) CSE and cognitive actions, and (2) CSE and metacognitive strategies Multiple linear
regression tests To measure the relative importance of CSE with regards to its contribution toward:
(1) cognitive actions, and (2) metacognitive strategies
Data Analysis
Answering Research Question 2 and Sub-question
Research
question How do students plan and monitor their cognitive actions, and regulate their monitoring strategies during learning with ILM?
Procedure Method Purpose
Repeated measures
To measure significant differences between: (1) planning and cognitive actions, (2) monitoring and cognitive
actions, and (3) monitoring and regulating strategies.
Cluster analysis To determine which screen-captured videos will be analyzed and to select students need to be interviewed.
Screen-captured
video analysis To explain findings from questionnaire analysis about how Cognitive Actions were planned and monitored, and how Monitoring Strategies were regulated.
Interview
analysis To explain findings from questionnaire and screen- captured video analysis.
Analyzing Screen-Captured Videos
Students’ interactions with ILM were captured & transcribed
List of events were transformed into
‘meaningful’
sequence of events
Coding process:
Cognitive actions, Planning, Monitoring, and Regulating Strategies
Graph that visualize ‘ dynamicity’ of strategy changes
and duration for each strategy
Frequency of strategy changes
Duration of strategies
1
2
3
4
Findings of the Study and Discussion
Demographic Information
One-hundred students (77 males and 23 females) completed all activities in this study during the spring 2013 semester.
About 62% of the participants had a GPA 3.00 or higher.
About 66% participants were considering majoring in a field of
engineering, technology, or computer science.
Data Homogeneity
Data from CSE and SRCBL questionnaires were used in the analysis to investigate whether differences existed among the participants.
The findings revealed that there was no significant difference between Logan and InTech Collegiate High Schools insofar as their computer self-efficacy, cognitive actions, planning,
monitoring, regulating, and overall metacognitive strategies.
In summary, these findings suggested that the data collected
from both schools were homogeneous.
Addressing Research Question #1
“How is students’ CSE related to cognitive actions and metacognitive strategies while using ILM?”
Descriptive Statistics
Beginning Skills Advanced Skills File and Software Skills
Computer Self-Effficacy
Planning Strategies
Cognitive Actions
Monitoring Strategies
Regulating Strategies
Cognitive Actions & Metacognitive Strategies
M= 4.54; SD= 0.52
M= 4.12; SD= 0.73
M= 4.34; SD= 0.64
M= 2.95; SD= 0.61
M= 2.72; SD= 0.59
M= 2.91; SD= 0.62
M= 2.89; SD= 0.54
* Low-to-moderate: Mean value between 1.00 and 2.75
* Moderate-to-high: Mean value between 2.76 and 4.00
* Terminology used in Lawanto, Butler, Cartier, Santoso, Lawanto, and Clark, 2013
Addressing Research Question #1
“How is students’ CSE related to cognitive actions and metacognitive strategies while using ILM?”
Relationships between CSE and Cognitive Actions and Metacognitive Strategies
A significant positive correlation between CSE and cognitive actions, r (100) = .176, p < .05
A significant positive correlation between advanced skills component of the CSE and cognitive actions, r (100) = .185, p < .05
No significant correlation between CSE and overall metacognitive strategies, r (100) = .121, p = .115
Significant positive relationships between CSE and planning strategies, r
(100) = .176, p < .05, and between beginning skills component of CSE and
planning strategies, r (100) = .186, p < .05
Addressing Sub-Question #1
The Relative Importance of CSE with Regards to Its Contribution toward Students’ Cognitive Actions and Metacognitive Strategies
To what degree CSE predicts Cognitive Actions: The three CSE components (i.e., beginning, advanced, and file and software skills) explained only 3.40% of the variance [R2 = .034, F(3, 96) = 1.115, p = .347] (*)
To what degree CSE predicts Planning Strategies: The three CSE components explained only 3.90% of the variance [R2 = .039, F(3, 96) = 1.302, p = .278] (*)
To what degree CSE predicts Monitoring Strategies: The three CSE components explained only 2.50% of the variance [R2 = .025, F(3, 96) = .837, p = .477] (*)
To what degree CSE predicts Regulating Strategies: The three CSE components explained negative 2.70% of the variance [R2 = -.027, F(3, 96) = .147, p = .931] (*)
(*) There are other factors that might contribute more to student cognitive actions, planning, monitoring, and regulating strategies while using ILM and should be investigated in future research.
Addressing Research Question #2
“How do students plan (PLA) and monitor (MON) their cognitive
actions, and regulate (REG) their monitoring strategies during learning with ILM?”
Cognitive Actions and Metacognitive Strategies of All Participants
A series of paired t tests (2-tailed) was conducted to evaluate whether gaps between SRL features were significant.
Gap Significant Difference? t and p values
PLA > COG Yes t = 5.967, p < .001
COG < MON Yes t = -5.418, p < .001
MON = REG No t = 1.036, p = .303
The students did well in making plan before working on the modules. But the findings show they might struggled to execute their plan. It may be caused by the objectives of the modules presented to them. Interestingly they performed well in monitoring their actions and regulating/adjusting strategies based on the monitoring process.
Addressing Sub-Question #2: Quantitative Analysis
Cognitive Actions & Metacognitive Strategies of High & Low CSE Groups
Cognitive Actions and Metacognitive Strategies between High (n = 47) and Low CSE (n = 16) Groups.
(Z = -2.176, p < .05)
(Z = -2.346, p < .05) (Z = -2.176, p < .05)
(Z = -2.972, p < .05) (Z = -2.546, p < .05)
Eight Selected Cases
Rationale: To gather a reasonable number of quantitative and qualitative data from the same subjects in investigating their cognitive actions and metacognitive strategies.
Selection Procedure:
Order screen-captured videos based on their duration
Prioritize the videos with long duration
Use a ‘stratified sampling’ to represent school and CSE ‘level’
Logan High School (n)
InTech Collegiate High School (n)
High CSE Student 2 2
Low CSE Student 2 2
Data ‘Triangulation’: Overview
Planning Str. * Cognitive Act. Monitoring Str.* Regulating Str.
High CSE 3.63 3.41 3.53 3.25
Low CSE 2.58 2.39 2.39 2.65
Planning Str. Cognitive Act. Monitoring Str. Regulating Str. * High CSE 30 min., 38 sec. 137 min., 3 sec. 56 min., 31 sec. 47 min., 28 sec.
Low CSE 35 min., 05 sec. 120 min., 13 sec. 47 min., 04 sec. 20 min, 19 sec.
From Quantitative Data [SRCBL Questionnaire]: Mean Score
From Qualitative Data [Screen-Captured Videos]: Duration of strategies
Planning Str. Cognitive Act. Monitoring Str. Regulating Str.
High CSE Similar: Read materials &
instructions first
Organized More elaborative Similar: Check progress first, then try other strategies.
Low CSE Trial & error Less elaborative
From Qualitative Data [Interview]: Issues Gathered
Note: * = significant difference
Note: * = significant difference
Addressing Sub-Question #2: Quantitative Analysis
Cognitive Actions & Metacognitive Strategies of High & Low CSE Selected Cases
Cognitive Actions & Metacognitive Strategies of High (n = 4) & Low (n = 4) Selected Cases.
(Z = .018, p < .05) (Z = .020, p < .05)
Both high and low CSE selected cases indicated no significant differences between PLA & COG, COG & MON, and MON & REG.
Addressing Sub-Question #2: Qualitative Analysis
Examples of Cognitive Actions & Metacognitive Strategies Profiles Case #1: Andy – High CSE
Case #5: Earl – Low CSE
COG: Matching the objects with ‘basic’ Boolean expressions PLA: Reading the learning materials
COG: Reading the learning materials
MON: Checking answer; Trying to review the guidance
REG: Trying other strategies
Addressing Sub-Question #2: Qualitative Analysis
Cognitive Actions and Metacognitive Strategies of Selected Cases
Frequency of Strategy Changes While Using the Modules between High and Low CSE Groups
The high CSE group changed their strategies more often that did the low CSE group on all modules.
Frequency of Strategy Changes per Group
Group Boolean Logic * Minimum Spanning Tree * Modeling Using Graphs *
High CSE Student 309 234 68
Low CSE Student 175 171 46
Frequency of Strategy Changes per Student
Group Boolean Logic * Minimum Spanning Tree Modeling Using Graphs
High CSE Student 77.25 58.5 17
Low CSE Student 43.75 42.75 11.5
Note: * = significant difference
Note: * = significant difference
Addressing Sub-Question #2: Qualitative Analysis Issues Gathered from Interviews
No. Issue Comparison
1. Previous experience in using a
computer helps students to use the interactive learning module.
Similar | previous experience helped them
2. Strategy of preparing to find solutions for the task.
Similar | read materials & instructions 3. Strategies to carry out plans while using
the ILM. Different | high CSE organized; low
CSE trial & error 4. Strategies used to detect any errors in
solving the task or problem.
Different | high CSE more
elaborative; low CSE less elaborative 5. Strategies to fix any errors in solving a
task or problem. Similar | see the error first 6. Success parameters of using the ILM
according to the students. Different | everyone has different perspectives
7. Aspects of ILM that students like and dislike the most.
Different | high CSE feedback mechanism; low CSE interface
Implications (1/2)
This research has implications for self-regulated learning researchers, teachers, and interactive learning module developers.
No CSE component significantly predicts cognitive actions and metacognitive strategies.
Self-regulated learning researchers may consider
identifying other factors or motivational constructs, such as intrinsic and extrinsic motivations, that may be significant predictors toward students’ cognitive actions and
metacognitive strategies while engaged in interactive
learning modules.
Implications (2/2)
It was indicated that the students found planning to be
important, but they did not translate the planning into actions.
The teachers may need to:
1. explain the introduction to the concepts before allowing the students to use the module;
2. encourage the students to read the objectives of activities on the modules carefully before executing their plans; and
3. have one or more teaching assistants to help him or her in
responding any question raised by students while working with the module.
Since the users’ computer self-efficacy is varied, the developers
should make the modules more easier to navigate and graphics
are created based on a real-world example.
Recommendations (1/2)
First, this study only analyzed 100 datasets of the participating students.
Larger participants from different schools may improve the generalizability of results of similar studies.
Second, the way high-school students work may influence the results.
The teacher presence during the data collection process could help the students to focus working on the modules.
Third, the nature of this study is descriptive study.
Experimental study can be conducted to see whether modified
interactive learning modules can improve either computer self-
efficacy or improve cognitive actions and metacognitive strategies.
Recommendations (2/2)
Fourth, a method used in this study may be beneficial for other studies in computer-based learning environments.
Fifth, while CSE questionnaire had high Cronbach’s Alpha scores, SRCBL questionnaire had relatively low reliability scores for
planning strategies.
Analyzing more than one type of interactive learning module or computer- based learning environment may help in improving the items of the
questionnaire.
Question and Answer
Email: [email protected]
