Table 3-1 shows the distribution of the approaches used in the sample of 32 articles.3 In terms of modal frequencies, the majority of the articles used within-school data that are not available for replication. Most of the studies
3 Two other instruments were not included in the 32 identified studies on effort but are used frequently enough in the literature on persistence and attention that they warrant mention here: the Learning Behaviors Scale (LBS) (McDermott et al., 1999) and the Preschool Learning Behaviors Scale (PLBS) (McDermott et al., 2000). Both the LBS and PLBS measure four dimensions of children’s learning behaviors (competence motivation, attitude toward learning, strategic/flexible learning, and persistent/attentive learning), with nine items measuring persistence/attention (e.g., “Tries but concentration soon fades,” “Easily distracted or seeks distraction,” and “Doesn’t stick to tasks”) (McDermott, 1999; McDermott et al., 2002). The LBS and PLBS were not included in this chapter because no studies were identified as using these instruments to measure effort per se, which was the main selection criterion for the articles.
used cross-sectional data on elementary or middle school students and employed multivariate statistics. A summary listing of the key features of the measures used in the sample of 32 studies is presented in Table 3-2.
Table 3-1. Approaches to studies of effort Study approach
Count of Studies Using this approach At what grade level is the construct measured?
Preschool 1
Elementary school 10
Middle school 9
High school 4
Multiple 8
What is the study design?
Cross-sectional 17
Longitudinal 15
What is the method of analysis?
Case study 2
Bivariate 4
Multivariate 23
Multilevel 3
Is the sample generalizable?
Sample of convenience (an existing intervention program) 1
Students identified as at-risk 0
Within school 9
Within district or region 19
Nationally representative 3
Can the study be replicated?
Data and survey are available 6
Questionnaire is available 12
No, neither data nor survey are available 14
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Persistence/
effort scale
Self-report NA 1 Middle and high school;
229 public school students in Turkey
Agbuga & Xiang (2008)
8 items
Content validity: Items based on previously developed instrument (Fincham et al., 1989; Xiang & Lee, 2002).
Construct validity: Exploratory factor analysis yielded a single factor in a pilot study of Turkish students. Established validity and reliability with US students.
Internal consistency reliability (α): .84 for the pilot study (Source: Agbuga & Xiang, 2008)
Post-task reflection
Self-report One item for each construct:
challenge, skill, absorption, timelessness, and effort
1 High school and college (n = 45)
Ainley et al. (2008) 5 items
Content validity: Items based on an interactive computer program used in other studies on interest.
Construct validity: Multivariate analyses conducted by Ainley et al.
(2008) found the index of flow based on absorption, timelessness, and effort successfully differentiated between students experiencing flow and nonflow students.
(Source: Ainley et al., 2008) Interest-
participation scale
Teacher report
NA 1 Elementary school; 790
students beginning in the 1st grade from the Baltimore City Public School System
Alexander et al. (1993)
5 items
Content validity: Items based on the Wave I (1976–77) National Survey of Children and selected to be consistent with Kohn and Rosman’s research (1972a, 1972b, 1974) as cited in Alexander et al. (1993).
Construct validity: Exploratory factor analysis supported interest- participation as one factor for full sample and for several subgroups.
Internal consistency reliability (α): .77 to .82 across years (Source: Alexander et al., 1993)
Engagement behaviors
Teacher report
Work habits, externalizing behaviors, adaptability
1 Elementary school; 790 students beginning in the 1st grade from the Baltimore City Public School System
Alexander et al. (1997)
9 items
Internal consistency reliability (α): .95 for work habits, .83 for externalizing behaviors, and .85 for adaptability behaviors; average correlation of the three components of the scale = .60
(Source: Alexander et al., 1997)
Involvement Self-report NA 1 Middle school; 297 public
school students
Berndt & Keefe (1995)
6 items
Content validity: Items based on a previously developed scale by Berndt and Miller (1990), as cited in Berndt & Keefe (1995).
Internal consistency reliability (α): .65 for the involvement scale Convergent and divergent validity: Negative correlation between involvement and disruption (−.49, p < .001): students with higher grades reported more involvement (r = .29, p < .001) and less disruption (r = .−.34, p < .001) and were rated by teachers as more involved (r = .67, p < .001) and less disruptive (r = −.46, p < .001).
(Source: Berndt & Keefe, 1995)
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Persistence/
effort scale
Self-report NA 1 Middle and high school;
229 public school students in Turkey
Agbuga & Xiang (2008)
8 items
Content validity: Items based on previously developed instrument (Fincham et al., 1989; Xiang & Lee, 2002).
Construct validity: Exploratory factor analysis yielded a single factor in a pilot study of Turkish students. Established validity and reliability with US students.
Internal consistency reliability (α): .84 for the pilot study (Source: Agbuga & Xiang, 2008)
Post-task reflection
Self-report One item for each construct:
challenge, skill, absorption, timelessness, and effort
1 High school and college (n = 45)
Ainley et al. (2008) 5 items
Content validity: Items based on an interactive computer program used in other studies on interest.
Construct validity: Multivariate analyses conducted by Ainley et al.
(2008) found the index of flow based on absorption, timelessness, and effort successfully differentiated between students experiencing flow and nonflow students.
(Source: Ainley et al., 2008) Interest-
participation scale
Teacher report
NA 1 Elementary school; 790
students beginning in the 1st grade from the Baltimore City Public School System
Alexander et al.
(1993)
5 items
Content validity: Items based on the Wave I (1976–77) National Survey of Children and selected to be consistent with Kohn and Rosman’s research (1972a, 1972b, 1974) as cited in Alexander et al. (1993).
Construct validity: Exploratory factor analysis supported interest- participation as one factor for full sample and for several subgroups.
Internal consistency reliability (α): .77 to .82 across years (Source: Alexander et al., 1993)
Engagement behaviors
Teacher report
Work habits, externalizing behaviors, adaptability
1 Elementary school; 790 students beginning in the 1st grade from the Baltimore City Public School System
Alexander et al.
(1997)
9 items
Internal consistency reliability (α): .95 for work habits, .83 for externalizing behaviors, and .85 for adaptability behaviors; average correlation of the three components of the scale = .60
(Source: Alexander et al., 1997)
Involvement Self-report NA 1 Middle school; 297 public
school students
Berndt & Keefe (1995)
6 items
Content validity: Items based on a previously developed scale by Berndt and Miller (1990), as cited in Berndt & Keefe (1995).
Internal consistency reliability (α): .65 for the involvement scale Convergent and divergent validity: Negative correlation between involvement and disruption (−.49, p < .001): students with higher grades reported more involvement (r = .29, p < .001) and less disruption (r = .−.34, p < .001) and were rated by teachers as more involved (r = .67, p < .001) and less disruptive (r = −.46, p < .001).
(Source: Berndt & Keefe, 1995)
continued
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Teacher Rating Scale of School Adjustment (TRSSA)
Teacher report
Cooperative participation, self-directedness
1 164 kindergartners in the Midwestern United States
Birch & Ladd (1997) 11 items
Content validity: Instrument developed in collaboration with participating teachers.
Construct validity: Factor analysis revealed separate subscales for cooperative participation and self-directedness.
Internal consistency (α): .92 (cooperative participation), .91 (self- directedness)
(Source: Birch & Ladd, 1997) Motivational
intention questions for effort
Self-report NA 1 Middle and high
school; 289 public school students in the Midwestern United States
Chase (2001) 1 item
Content validity: Items based on previous research on intended effort and self-efficacy.
(Source: Chase, 2001)
Engagement in school
Self-report Behavioral engagement, emotional engagement
1 High school; 2,672 African American students in Atlanta, Baltimore, DC, and New York; from three independent data sets
Connell et al. (1994)
14 items
Content validity: Survey items were extracted from several existing instruments.
Construct validity: Items were selected based on results of internal consistency analyses.
(Source: Connell et al., 1994) Time spent on
homework
Self-report NA 1 High school; nationally
representative sample of sophomore students (n = 6,737)
DeLuca & Rosenbaum (2001)
1 item
Content validity: Drawn from the US Department of Education’s National Education Longitudinal Study of 1988 (NELS:88) data set.
(Source: DeLuca & Rosenbaum, 2001) Intellectual
Achievement Responsibility Scale (Crandall et al., 1965, as cited in Fincham et al., 1989)
Teacher report
Learned helplessness scale
1 Elementary school;
87 3rd graders in the Midwestern United States; with follow-up at 5th grade
Fincham et al. (1989)
10 items
Internal consistency reliability (α): .66 (3rd grade) and .54 (5th grade) (Source: Fincham et al., 1989)
Engagement Student report;
teacher report
2 Middle and high school;
nationally representative sample from the NELS:88 data set (n = 18,307 in Finn, 1993; and n = 1,803 minority students from low-income homes in Finn & Rock, 1997)
Finn (1993); Finn & Rock (1997)
4 items
Content validity: Items were drawn from the US Department of Education’s NELS:88 data set.
Construct validity: Principal component factor analysis was used to create scales.
(Source: Finn, 1993) (continued)
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Teacher Rating Scale of School Adjustment (TRSSA)
Teacher report
Cooperative participation, self-directedness
1 164 kindergartners in the Midwestern United States
Birch & Ladd (1997) 11 items
Content validity: Instrument developed in collaboration with participating teachers.
Construct validity: Factor analysis revealed separate subscales for cooperative participation and self-directedness.
Internal consistency (α): .92 (cooperative participation), .91 (self- directedness)
(Source: Birch & Ladd, 1997) Motivational
intention questions for effort
Self-report NA 1 Middle and high
school; 289 public school students in the Midwestern United States
Chase (2001) 1 item
Content validity: Items based on previous research on intended effort and self-efficacy.
(Source: Chase, 2001)
Engagement in school
Self-report Behavioral engagement, emotional engagement
1 High school; 2,672 African American students in Atlanta, Baltimore, DC, and New York; from three independent data sets
Connell et al.
(1994)
14 items
Content validity: Survey items were extracted from several existing instruments.
Construct validity: Items were selected based on results of internal consistency analyses.
(Source: Connell et al., 1994) Time spent on
homework
Self-report NA 1 High school; nationally
representative sample of sophomore students (n = 6,737)
DeLuca &
Rosenbaum (2001) 1 item
Content validity: Drawn from the US Department of Education’s National Education Longitudinal Study of 1988 (NELS:88) data set.
(Source: DeLuca & Rosenbaum, 2001) Intellectual
Achievement Responsibility Scale (Crandall et al., 1965, as cited in Fincham et al., 1989)
Teacher report
Learned helplessness scale
1 Elementary school;
87 3rd graders in the Midwestern United States; with follow-up at 5th grade
Fincham et al.
(1989)
10 items
Internal consistency reliability (α): .66 (3rd grade) and .54 (5th grade) (Source: Fincham et al., 1989)
Engagement Student report;
teacher report
2 Middle and high school;
nationally representative sample from the NELS:88 data set (n = 18,307 in Finn, 1993; and n = 1,803 minority students from low-income homes in Finn & Rock, 1997)
Finn (1993); Finn &
Rock (1997)
4 items
Content validity: Items were drawn from the US Department of Education’s NELS:88 data set.
Construct validity: Principal component factor analysis was used to create scales.
(Source: Finn, 1993)
continued
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Student Participation Questionnaire
Teacher report
Minimal adequate effort scale, initiative taking scale, and inattentive behavior scales
2 Elementary school;
approximately 2,000 4th graders in Tennessee
Finn et al. (1991); Finn et al. (1995)
21 items
Internal consistency reliability (α): .93 (minimal adequate effort), .89 (initiative taking), and .75 (inattentive behavior)
(Source: Finn et al., 1995)
Academic effort scale
Teacher report
NA 1 Elementary school
students; 437 students in the Northeastern United States
Gest et al. (2008) 5 items
Content validity: Items were drawn from well-validated rating scales.
Construct validity: Scales were derived from factor analysis.
Internal consistency reliability (α): Ranged from .92 to .93 across five separate student assessments.
(Source: Gest et al., 2008) Task persistence Researcher
observation
Task directed scale
1 Preschool and elementary school students; 43 toddlers in Australia
Gilmore et al. (2003)
4 items
Content validity: Persistence in challenging tasks was coded using procedures developed by others.
(Source: Gilmore et al., 2003) Effort/
persistence scale
Student report
Effort, persistence
1 High school; 544 high school students from the Southwestern United States
Guan et al. (2006) 8 items
Content validity: Items were adapted from other measures.
Construct validity: Principal component factor analysis resulted in a single factor.
Internal consistency reliability (α): .90 (Source: Guan et al., 2006)
School engagement and effort scale
Student report
NA 1 High school; 900 public
high school students in the rural Southwestern United States
Hardre et al. (2007) 6 items
Content validity: Items from existing instruments were included.
Internal consistency reliability (α): .85
Construct validity: Confirmatory factor analysis resulted in a single factor for the school engagement scale.
(Source: Hardre et al., 2007) Rochester
Assessment of Intellectual and Social Engagement
Student report
Effort 1 Middle school; 373
students in the Midwestern United States
Kiefer & Ryan (2008)
5 items
Internal consistency reliability (α): .77 (Year 1) and.80 (Year 2)
Concurrent validity: Concordance between student and teacher reports of student effort
Test-retest reliability: Effort was fairly stable across the transition to middle school from 6th to 7th grade (r = .42).
(Source: Kiefer & Ryan, 2008) Engagement vs.
disaffection
Teacher report
NA 1 Middle school; an entire
cohort of 366 6th graders attending the only public middle school available to a Northeastern US town
Kindermann (2007) 14 items
Content validity: Items were drawn from previously established instrument.
Concurrent validity: Engagement moderately correlated with grades and achievement scores (ranging from .40 in mathematics achievement to .58 in reading).
Test-retest reliability: Modified items have reported high stability (r = .73, p < .00, n = 144, over an 8-month period).
(Source: Kindermann, 2007) (continued)
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Student Participation Questionnaire
Teacher report
Minimal adequate effort scale, initiative taking scale, and inattentive behavior scales
2 Elementary school;
approximately 2,000 4th graders in Tennessee
Finn et al. (1991);
Finn et al. (1995)
21 items
Internal consistency reliability (α): .93 (minimal adequate effort), .89 (initiative taking), and .75 (inattentive behavior)
(Source: Finn et al., 1995)
Academic effort scale
Teacher report
NA 1 Elementary school
students; 437 students in the Northeastern United States
Gest et al. (2008) 5 items
Content validity: Items were drawn from well-validated rating scales.
Construct validity: Scales were derived from factor analysis.
Internal consistency reliability (α): Ranged from .92 to .93 across five separate student assessments.
(Source: Gest et al., 2008) Task persistence Researcher
observation
Task directed scale
1 Preschool and elementary school students; 43 toddlers in Australia
Gilmore et al.
(2003)
4 items
Content validity: Persistence in challenging tasks was coded using procedures developed by others.
(Source: Gilmore et al., 2003) Effort/
persistence scale
Student report
Effort, persistence
1 High school; 544 high school students from the Southwestern United States
Guan et al. (2006) 8 items
Content validity: Items were adapted from other measures.
Construct validity: Principal component factor analysis resulted in a single factor.
Internal consistency reliability (α): .90 (Source: Guan et al., 2006)
School engagement and effort scale
Student report
NA 1 High school; 900 public
high school students in the rural Southwestern United States
Hardre et al. (2007) 6 items
Content validity: Items from existing instruments were included.
Internal consistency reliability (α): .85
Construct validity: Confirmatory factor analysis resulted in a single factor for the school engagement scale.
(Source: Hardre et al., 2007) Rochester
Assessment of Intellectual and Social Engagement
Student report
Effort 1 Middle school; 373
students in the Midwestern United States
Kiefer & Ryan (2008)
5 items
Internal consistency reliability (α): .77 (Year 1) and.80 (Year 2)
Concurrent validity: Concordance between student and teacher reports of student effort
Test-retest reliability: Effort was fairly stable across the transition to middle school from 6th to 7th grade (r = .42).
(Source: Kiefer & Ryan, 2008) Engagement vs.
disaffection
Teacher report
NA 1 Middle school; an entire
cohort of 366 6th graders attending the only public middle school available to a Northeastern US town
Kindermann (2007) 14 items
Content validity: Items were drawn from previously established instrument.
Concurrent validity: Engagement moderately correlated with grades and achievement scores (ranging from .40 in mathematics achievement to .58 in reading).
Test-retest reliability: Modified items have reported high stability (r = .73, p < .00, n = 144, over an 8-month period).
(Source: Kindermann, 2007)
continued
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Effort withdrawal scale
Student report
NA 1 Middle school; 3,943
5th graders
Lau & Nie (2008) 4 items
Content validity: Items based on research investigating student reports of attention, effort, and participation in math classes.
Construct validity: Confirmatory factor analysis revealed a single factor for effort withdrawal.
Internal consistency reliability (α): .80 (withdrawal scale) (Source: Lau & Nie, 2008)
Task engagement
Researcher observation
NA 1 Middle school; 130 total
observations of 12 6th grade students from two middle school classrooms in an urban Midwestern US school district
Lee & Anderson (1993)
3 codes
Content validity: Codes for qualitative data were based on two conceptual frameworks from existing classroom research (student motivation to learn and conceptual change in science).
Inter-rater reliability: 81% between two coders (Source: Lee & Anderson, 1993)
Engagement scale
Student report
One item for each component:
effort, attentiveness, lack of boredom in class, and completing class assignments
1 3,669 5th, 8th, and 10th graders attending 24 schools that made
“substantial progress in restructuring” (n = 3,699)
Marks (2000) 4 items
Internal consistency reliability (α): .69
Concurrent validity: Correlated with classroom observations on student engagement (r = .37).
(Source: Marks, 2000)
Task-specific effort
Student report
NA 1 Middle school; 102
African American and Latino students
Roderick & Engel (2001)
4 codes
No information available
Academic effort Teacher report
NA 1 329 middle school
students in the Midwestern United States
Rudolph et al. (2001)
1 item
No information available
Flow scale via experience sampling method
Student report
NA 1 Middle school; 165
8th graders in the Midwestern United States
Shernoff & Vandell (2007)
3 items
No information available
Student engagement and disaffection
Teacher report
NA 1 200 middle school
students in upstate New York
Skinner & Belmont (1990)
10 items
Split-half reliability: r = .82 (Source: Skinner & Belmont, 1990) (continued)
Table 3-2. Measures of effort: Key features
Measure Name Data Source
Subscales or Components
No. of Studies
Using this
Measure Intended population example articles psychometric properties
Effort withdrawal scale
Student report
NA 1 Middle school; 3,943
5th graders
Lau & Nie (2008) 4 items
Content validity: Items based on research investigating student reports of attention, effort, and participation in math classes.
Construct validity: Confirmatory factor analysis revealed a single factor for effort withdrawal.
Internal consistency reliability (α): .80 (withdrawal scale) (Source: Lau & Nie, 2008)
Task engagement
Researcher observation
NA 1 Middle school; 130 total
observations of 12 6th grade students from two middle school classrooms in an urban Midwestern US school district
Lee & Anderson (1993)
3 codes
Content validity: Codes for qualitative data were based on two conceptual frameworks from existing classroom research (student motivation to learn and conceptual change in science).
Inter-rater reliability: 81% between two coders (Source: Lee & Anderson, 1993)
Engagement scale
Student report
One item for each component:
effort, attentiveness, lack of boredom in class, and completing class assignments
1 3,669 5th, 8th, and 10th graders attending 24 schools that made
“substantial progress in restructuring” (n = 3,699)
Marks (2000) 4 items
Internal consistency reliability (α): .69
Concurrent validity: Correlated with classroom observations on student engagement (r = .37).
(Source: Marks, 2000)
Task-specific effort
Student report
NA 1 Middle school; 102
African American and Latino students
Roderick & Engel (2001)
4 codes
No information available
Academic effort Teacher report
NA 1 329 middle school
students in the Midwestern United States
Rudolph et al.
(2001)
1 item
No information available
Flow scale via experience sampling method
Student report
NA 1 Middle school; 165
8th graders in the Midwestern United States
Shernoff & Vandell (2007)
3 items
No information available
Student engagement and disaffection
Teacher report
NA 1 200 middle school
students in upstate New York
Skinner & Belmont (1990)
10 items
Split-half reliability: r = .82 (Source: Skinner & Belmont, 1990)
continued