Two Multi-Site Randomized Control Trials:
Bottom-Up vs. Top-Down Approaches to Scaling Up PALS
Is Response to Word-Problem
Intervention among Students with MD Moderated by Concurrent RD?
Doug Fuchs and Lynn Fuchs Vanderbilt University
Bottom-Up vs. Top-Down Approaches to Scaling Up PALS
Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen,
Don Compton, and Chris Lemons Vanderbilt University
Chris Schatschneider Florida State University
R305G04104
Institute of Education Sciences
Purpose of PALS
Supplements the general education core program
Implemented 3 times per week in reading; 2 times per week in math
Creates a “routine” for teachers to differentiate instruction by creating many simultaneous peer- mediated lessons rather than one teacher-directed lesson
PALS Reading: kindergarten, first grade, grades 2-6, high school
PALS math: kindergarten, first grade, grades 2-6
PALS Research
• Based on Juniper Gardens Classwide Peer Tutoring
• Over 15 years of experimental research
• Title I and Non-Title I schools
• Urban and suburban schools
• High, average, and low achievers
• Students with learning disabilities
• “Validated Practice” status (USDE, WWC, BEE)
Grades 2-6 PALS
Partner Reading
Paragraph Shrinking
Prediction Relay
Partner Reading
• Conducted for 11-12 minutes
• Stronger reader reads aloud for 5 minutes
• Weaker reader reads same text aloud for 5 minutes
• Weaker reader retells story for 1-2 minutes
• Readers read quickly, correctly, and with expression
• Coaches listen, correct mistakes, and mark points (1 point for each correctly read sentence and 10 points for story retell)
Paragraph Shrinking
• Conducted for 10 minutes
• For 5 minutes:
• Stronger reader reads new text aloud, summarizing paragraph by paragraph
– Name the most important who or what (1 point) – Name the most important thing about the who or
what (1 point)
– Shrink it to 10 or fewer words (1 point)
• For next 5 minutes:
• Weaker reader reads new text aloud, summarizing paragraph by paragraph (as above)
• Coach listens, corrects mistakes, and marks points
Prediction Relay
5 minutes, stronger reader read new text
− Makes prediction (1)
− Reads half page (1)
− Checks prediction (1)
− States main idea (3)
− Makes new prediction
− Continues to read
5 minutes, weaker reader continues on in new text, with the same activities
Coach listens, corrects mistakes, and marks
points
Two Kinds of PALS Research
Randomized Controlled Trials
Study schools include Title I and no Title I.
Classrooms randomly assigned within schools to PALS and control groups.
HA, AA, LA (including LD) students targeted in each classroom.
Fidelity of treatment implementation.
Individually administered pre-/posttests by
0 20 40 60 80
Learning Disabled
Low Achieving
Average Achieving
High Achieving Experimental Control
Improvement Over 16Weeks
Improvement in Reading
Two Kinds of PALS Research
School-Improvement Projects
Title I schools implement PALS school-wide.
Our TA funded by Nashville’s Title I office No fidelity of treatment assessed.
Group administered high-stakes tests.
Report Card Scores Based on Students’
Performance on the TCAP
(CTB/McGraw-Hill)GOWER
MATH 76.0 (61) 107.8 (12)
‘93 ‘94
Subject
Language Arts 61.7 (60) 91.8 (33) READING 74.0 (60) 112.5 (25)
Science 74.4 (58) 95.6 (24) Social Studies 60.1 (61) 81.4 (53)
Note 1: A score of 100 means that students of a school are progressing at a rate equivalent to that of the national rate.
Typical Teacher Support in PALS Research
Support was ongoing and on-site: RAs gave in-class assistance 1x or 2x per wk during training and
implementation.
RA support is costly, unlikely in wider implementations, an obstacle to scaling up.
Absent such support, quality of implementations suffer.
How to separate an intervention from its support
system? How to scale-up (“export” the intervention from A to B) without researchers’ nurturance?
There’s also a matter of transcending time….
Purpose of the Present Study
With Grades 2-6 PALS as a “prop,” and students’ reading achievement as the criterion, does a “bottom-up” approach
beat a “top-down” approach to scaling up?
Do these approaches affect teachers’
sustainability of PALS?
METHOD
Participating Sites
Vanderbilt University (Nashville)
Where K-PALS was developed and
researched Has K-PALS
“expertise”(Project Director, Coordinators)
“PALS” has brand- name recognition
University of Minnesota (Twin Cities
Area)
Large urban area, diverse student
population Schools had some
prior experience with PALS Schools regularly
participate in University research
The University of Texas Pan
American (Hidalgo
County)
Southern-most region of TX along
the U.S. Mexico Border Schools had no prior experience
with PALS Schools had no prior experience
participating in
Participating Teachers
Two cohorts of 3
rd-, 4th-, and 5th-grade teachers :
− Cohort 1
Entered study in 2006-07
− Cohort 2
Entered study in 2007-08
Two years of study participation:
− Year 1
Assigned randomly to PALS or Control
− Year 2
PALS Teachers selected Top Down or Bottom Up PALS
Control Teachers remained in Control group
Teachers by Study Group in Year 2
Top Down Bottom Up Control
Cohort 1
Tennessee 5 5 7
Minnesota 5 5 4
Texas 6 7 7
Total 16 17 18
Cohort 2
Tennessee 12 8 9
Minnesota 6 9 12*
Texas 8 4 7
Total 26 21 28*
TOTAL
Teachers 42 38 46*
Students by Study Group in Year 2
Top Down Bottom Up Control
Cohort 1
Tennessee 60 60 68
Minnesota 61 65 49
Texas 61 90 88
Total 182 215 205
Cohort 2
Tennessee 136 86 102
Minnesota 72 109 151*
Texas 95 46 73
Total 303 241 326*
TOTAL
Students 485 456 531*
Study Conditions: Year 1
Control – Teachers implemented core language arts curriculum
PALS – Teachers implemented with fidelity:
− 3 times/week for 35-40 min (about 54 sessions)
− Coaches and Readers: higher-performing readers paired with lower-performing readers
− Four PALS Activities:
Partner Reading (10 min)
Retell (2 min)
Paragraph Shrinking (10 min)
Prediction Relay (10 min)
Study Conditions: Year 2
Control – Teachers implemented core language arts curriculum
“Top Down” (TD) PALS
− Teachers did PALS “by the book”
− Fidelity of PALS implementation was emphasized
“Bottom Up” (BU) PALS
− Teachers implemented core components of PALS
− Customization was strongly encouraged and supported
BU PALS: Core Elements
48 sessions minimum
35 minutes per session minimum
10 minutes of Partner Reading
10 minutes of Paragraph Shrinking
A motivational peer reinforcement system
BU PALS: Requirements
Teachers asked to:
− Conduct core elements of PALS as designed
− Develop changes
Match to curriculum, students’ needs, teaching style
Create a type of PALS for the long term
Student Measures
Academic Measures
Test of Word Reading Efficiency (TOWRE)
Woodcock Reading Mastery Test- Revised (WRMT-R)
− Word Identification Subtest
Wide Range Achievement Test (WRAT)
− Letter and Word Identification Subtests
Comprehensive Reading Assessment Battery (CRAB; 2 passages)
− Oral reading (1 min & 3 min)
− Comprehension (10 open-ended questions)
CBM Maze Task (2 passages)
− Correct maze choices made in 2.5 min
Iowa Test of Basic Skills (ITBS)
− Reading Comprehension
− Vocabulary
Student Characteristics
Demographics
SWAN
− Teachers rated each
student’s abilities to focus attention, control activity, and inhibit impulses
Teacher ratings
− Teachers rated each
student’s effort in reading and behavior in the
classroom
Teacher Measures
Classroom Measures
PALS Calendars
PALS Fidelity
Language Arts Observation
Classroom Atmosphere Rating Scale (Wehby)
Survey of Enacted Curriculum (SEC): English and Language Arts
Teacher Characteristics
Demographics
Berends teacher survey (assesses school climate, teacher professional
development, teacher efficacy, etc.)
Procedures
Pretesting (September-October)
PALS Workshops (September-October)
− Year 1: All teachers attend same workshop
− Year 2: Separate TD and BU workshops
PALS Implementation (~18 weeks)
− Teachers implemented 3 times per week for 35-40 min
− Weekly classroom visits from project staff
− Three “booster” sessions for TD and BU PALS teachers
− Two fidelity observations
Language arts observations in PALS and Control classrooms
− 45-60 min
− Momentary time sampling of a variety of reading instructional components
− Supplementary field notes
Posttesting (March-May)
Scaling-Up PALS for Grades 2-6
Results
Organization of Study
COHORT 1 (1st YEAR)
Top-Down PALS (W + B +
Tutor)
Top-Down PALS (W + B + Helpers)
Control Year 1 (2006-07)
Year 2 (2007-08)
Top-Down PALS
Control
Year 3 (2008-09)
COHORT 2 (2nd YEAR)
Top-Down PALS
Bottom-Up PALS
Control COHORT 1
(2nd YEAR)
Control Bottom-Up PALS Top-Down PALS
COHORT 2 (1st YEAR) COHORT 1
(1st YEAR)
Top-Down PALS (W + B +
Tutor)
Top-Down PALS (W + B + Helpers)
Control Year 1 (2006-07)
Year 2 (2007-08)
Top-Down PALS
Control
Year 3 (2008-09)
COHORT 2 (2nd YEAR)
Top-Down PALS
Bottom-Up PALS
Control COHORT 1
(2nd YEAR)
Control Bottom-Up PALS Top-Down PALS
COHORT 2 (1st YEAR)
Analysis Procedures
Create latent pretest and posttest variables combining 5 reading measures into 1
Create a latent change score
− Produces an “error-free” change value
Run 2-level HLM analyses
− Outcome: Latent change score
− Variables: Treatment condition (TD, BU, Control); Site (TN, MN, TX); latent pretest score
− Random effects: Level 2 teacher effects; ICC = .10
Test comparability of groups on variables plausibly related to selection of TD or BU
Descriptive Statistics
Regression Analysis
Effects for Study Groups by LA,
AA, and HA Students
Effects for Study Groups by LA, AA, and HA Students
*
*
*
Is Response to Word-Problem
Intervention among Students with MD Moderated by Concurrent RD?
Lynn Fuchs, Sarah Powell, Pamela Seethaler, Paul Cirino, Jack Fletcher, Doug Fuchs,
Carol Hamlett, and Rebecca Zumeta
Vanderbilt University and University of Houston Journal of Educational Psychology, 2009
Grant #P01046261 National Institute of
Child Health and Human Development
Study Purposes
Examine the efficacy of tutoring protocols for remediating
− Math fact deficits
− Word problem deficits
Assess whether treatment efficacy is different for
− Students with MD alone versus
− Students with MDRD
Determine whether effects are comparable as a function of site
− Nashville, where the tutoring protocols were developed
− Houston, a site distal to the developers
Participants
924 students screened in 63 classrooms in 18 schools in Nashville and Houston (similar sample size at each site)
Inclusion criteria:
− WRAT-A: < 26th percentile
− 5-item word-problem measure: score < 2
− At least 1 (of 2) WASI subtest T score: > 36
162 students eligible for the study; 133 students remained at posttesting
Blocking on site (Nashville and Houston) and MD status (MD vs. MDRD), students randomly assigned to tutoring conditions:
− Math Facts Tutoring (“Math Flash”)
− Word Problem Tutoring (“Pirate Math”)
− Control
Participants
Treatment groups comparable on all variables
MD vs. MDRD differences (across treatment groups) as expected
MD MDRD
Age 9 9
Female 40% 48%
Sub. Lunch 68% 90%
Spec. Ed. 8% 28%
WASI IQ 92 85
WRAT-A 88 81
WRAT-R 105 78
Examined Efficacy
of Two Tutoring Protocols
Both Tutoring Protocols
Delivered individually
48 sessions: 3 per week for 16 weeks
20-30 minutes per session
Scripted lessons, which tutors studied (not read)
Motivational system to ensure on-task behavior and hard, accurate work
Each session audiotaped; tapes sampled and coded for fidelity, which was high for both tutoring conditions
Examined Efficacy
of Two Tutoring Protocols
The exclusive focus of Math Flash was math facts
The primary focus of Pirate Math was word problems
− but it also addressed foundational skills
(math facts, procedural calculations, and
algebra skills)
Pirate Math Tutoring
48 sessions: 3 per week for 16 weeks 20-30 minutes per session
Scripted lessons, which tutors study (not read) Four units
Foundational Skills for Word Problems Total Word Problems
Difference Word Problems
Change Word Problems
Pirate Math: Introductory Unit
Teach students:
− Efficient counting strategies to answer math facts
− 2-digit procedural calculations
− How to solve for X in addition and
subtraction equations (a+b=c; x-y=z)
− How to check work
Introductory Unit:
Counting Up
Introductory Unit:
Finding X in All 3 Positions of Equation
If X is at the end of a number sentence, do what the problem tells you to do (e.g., 3 + 2 = X; 6 – 2 = X)
If X is not at the end, and it’s an “X minus”
problem, add (e.g., X – 2 = 4).
If X is not at the end, and it’s not a X
minus problem, subtract (e.g., X + 2 = 8; 5
– X = 2; 7 + X = 12).
Introductory Unit:
Checking Work
Remaining Units:
Word-Problem Lessons
Following Unit 1, four activities per session.
1. Flash-card warm up
2. Conceptual/strategic lesson using schema-broadening instruction
3. Sorting practice on identifying problem types 4. Paper/pencil review
1. Math Fact Flash Card Warm Up
Math Fact flash cards comprise 200 addition and subtraction facts
− Sums 0-18
− Subtrahends 0-18
Tutor shows flash card to student: Know it or Count Up!
− If student answers correctly, flash card placed in correct pile.
− If student answers incorrectly, tutor asks student to
“Count Up”; once correct, goes in correct pile.
− Student graphs score on graph.
4 + 5
11 - 6
2. Lesson
Pirate Math RUN
Students use
“RUN” strategy for every word
problem.
Students learn to circle relevant
information
directly in the text or
picture/graph/chart
2. Lesson
Pirate Math Setting Up Work
Write the equation that goes with the problem type.
Figure out what’s missing. Write X in your equation in the appropriate slot.
Figure out what numbers are known. Write those numbers in the appropriate slots.
Write the math signs.
Find X.
Make sure your answer has a number and a label.
50
2. Lesson
Problem Types with Transfer
Problem types at grade 2: Total, Difference, and Change
Transfer features:
− Irrelevant information
− Money
− Double-digit calculations
− Finding relevant information in graphs and pictures.
51
2. Lesson
Pirate Math Change
Change problems with a starting
amount that increases or decreases (a
change) to make it a new amount.
“Sarah had 10
pencils. Then she gave 4 pencils to Pamela. How many pencils does Sarah have now?”
CHANGE
1. How many do you start with? (St) 2. How many do you change? (C)
Is there an increase? + Is there a decrease? -
3. How many do you end with? (E)
St + or - C = E
4. Write the number sentence.
5. Find X!
“Sarah had 10 pencils. Then, she gave 4 pencils to Pamela. How many pencils does Sarah have
now?”
“Sarah had 10 pencils. Then, she gave 4 pencils to Pamela. How many pencils does Sarah have
now?”
Recognize problem type: Change problem.
Write equation for Change problems: St +/- C = E.
Identify missing information (E). Write that in the appropriate slot
St +/- C = E X
Identify the important given numbers (St and C). Write those in the appropriate slots.
St +/- C = E 10 4 X
Write math signs.
St +/- C = E 10 - 4 = X
Find X: X is at end so do what problem tells me to do: 10 – 4 = 6; X=6.
Label answer: 6 pencils.
Lexie had some comic books in her desk. Then she bought 8 more. Now, she has 12 comic books.
How many comic books did Lexie have in her desk to begin with?
Lexie had some comic books in her desk. Then she bought 8 more. Now, she has 12 comic books.
How many comic books did Lexie have in her desk to begin with?
Recognize problem type: Change problem.
Write equation for Change problems: St +/- C = E.
Identify missing information (St). Write that in the appropriate slot
St +/- C = E X
Identify the important given numbers (St and C). Write those in the appropriate slots.
St +/- C = E X 8 12 Write math signs.
St +/- C = E X + 8 = 12
Find X: X is not at end and it’s not an X minus problem, so subtract: 12 – 8 = 4; X=4.
Alicia has 3 friends in her math class.
The chart shows how many stars Alicia and her friends earned on Monday. On Tuesday, Alicia lost 3 stars for talking. How many stars
does she have now?
Monday’s Star Chart
0 2 4 6 8 10
David Trish Milo
Alicia
Recognize problem type: Change problem.
Identify transfer features: Irrelevant information (cross it out) and relevant information in a graph (number the graph).
Write equation for Change problems: St +/- C = E.
Identify missing information (St). Write that in the appropriate slot
St +/- C = E
X
Identify the important given numbers (St and C). Write those in the appropriate slots.
St +/- C = E 8 3 X
Write math signs.
St +/- C = E 8 - 3 = X
Find X: X is at end, so do what the problem says: 8 – 3 = 5;
2. Lesson
Pirate Math Total
Total problems have two parts that are combined for a total.
Total amount is the entire or combined amount.
“Sarah has 5 pencils.
Pamela has 3 pencils.
How many pencils do the girls have in all?”
P1 + P2 = T
2. Lesson
Pirate Math Difference
Difference problems compare two amounts to find the difference between them.
“Sarah has 7 pencils.
Pamela has 12
pencils. How many more pencils does Pamela have than Sarah?”
B – s = D
3. Sorting
Student sorts word problems by problem type for 2 minutes.
Tutor reads cards to student.
Student places cards on Sorting Mat.
At end of 2 minutes, tutor counts number of correctly sorted cards and uses
correction procedure for up to 3 incorrectly
sorted cards.
3. Sorting
3. Sorting
Maria and Jackie picked 16 flowers. Jackie picked 7 flowers. How many flowers did Maria
pick?
Maria picked 8 more flowers than Jackie.
Jackie picked 4 flowers.
How many flowers did Maria pick?
Maria picked 11 flowers.
Then Jackie took 4 of them for her Mom. How
many flowers does
4. Paper/pencil review
* 10 math facts
* 4 double-digit calculations
* 1 word problem
Motivation during Pirate Math
Students earn coins throughout lesson for
listening well, working hard, following directions, and
correct work.
At end of lesson, students color footsteps on treasure map equaling amount of coins earned.
When students color 16 footsteps, they pick a prize from treasure box and
receive a new map.
Examined Efficacy
of Two Tutoring Programs
The exclusive focus of math facts tutoring was math facts
The primary focus of Pirate Math tutoring was word problems
− but it also addressed foundational skills
(math facts, procedural calculations, and
algebra skills)
Efficacy:
Fluency with Math Facts and Procedural Calculations
On math facts, Pirate Math effects superior
improvement compared to control group. No difference between Pirate Math and math facts tutoring. Notable, because Pirate Math only allocates an initial lesson and then 4-6 minutes per session on number combinations.
On procedural calculations, Pirate Math effects superior improvement compared to control group and compared to math facts tutoring. Again, little time spent on
procedural calculations.
Efficacy : Algebra
On algebra, Pirate Math effects superior outcomes
compared to control group and compared to math facts tutoring.
Algebraic cognition improved even though students were severely deficient in math and young.
Given strong focus on algebra in high schools, given graduation requirements for algebra, and given
emphasis in NMAP, introducing algebra earlier in the curriculum may represent a productive innovation.
Correct Representation
Correct Representation
Incorrect Representation
Over Efficacy Results: Word Problems
Work on these foundation skills (MFs, procedural calculations, algebra), combined with schema-broadening instruction, also produced differential growth on WP outcomes compared to MF tutoring group and compared to control group.
MF tutoring did not result in improvement on WPs.
Lack of transfer suggests that source of difficulty is not diverting attention from the complex mathematics to the MFs embedded in those problems, but rather failing to comprehend the relations
among the numbers embedded in the narratives or to process the language in those stories adequately.
Suggests that MFs may not be the bottleneck for WP performance.
Instead, mathematics disability represents a more complicated pattern of difficulty, implicating language (as has been suggested elsewhere).
Is Tutoring Differentially Efficacious
Depending on MD Status (MD vs. MDRD)?
Why We Hypothesized MD students would be more responsive to tutoring than MDRD students
For MFs
− A key deficit among students with reading difficulty is phonological processing and
− Phonological processing deficits are linked with difficulty in automatic retrieval of MFs.
For WPs
− Using text to construct a WP model involves language
− Language profiles of students with MDRD are depressed compared to students with MD.
Is Tutoring Differentially Efficacious
Depending on MD Status (MD vs. MDRD)?
No evidence of differential responsiveness to
intervention as a function of difficulty status on any outcome.
Raises questions about the tenability of the MD/MDRD subtyping scheme and suggests the need to pursue other avenues for subtyping mathematics disability.
Even so, across tutoring conditions and sites, students with MD outperformed students with MDRD at pre- and posttest.
Additional work to examine the tenability of the
MD/MDRD subtyping scheme is warranted, even as
These Tutoring Protocols Are Transportable
No MD/MDRD by treatment by site effects.
No treatment by site effects.
Tutoring protocols were comparably
effective in Nashville and Houston, for MD
and for MDRD students.
Overall Conclusions
MF tutoring enhances fluency with MFs
with transfer to procedural calculations but without transfer to algebra or WPs.
