Pair-wise Comparisons

ANOVA MPS COMP

4.1.2 Pair-wise Comparisons

Tables 4.8 through 4.11 present four years (`97, '98, '99, and 2000) of state assessment results for each of the 17 SOI schools versus their matched comparison schools. Readers will recall that matched comparison schools were selected using variables like school size and location, school SES ranking, and school performance on state assessments in reading/literature and math at grades 3 and 5. The tables are presented in order by grade and academic subject. Also presented in each table are Oregon's statewide averages for three school years and this year's group average for the 17 SOI schools and their matched counterparts.

Table 4.8 shows four years of state assessment data for grade 3 reading/literature. Points worthy of note in this table include:

1. For 2000, 13 of 17 SOI schools bettered their own previous year's performance; 12 of 17 comparison schools bettered their own previous year's performance.

2. Ten of 17 SOI schools and a similar number of comparison schools showed a relatively stable or improving trend in 3rd grade reading scores over the four years reported.

3. Of the 2 schools participating in the SOI Program for a third year, one showed improvement over the previous year, to return to its 1998 average (Gray); and one showed a small increase from the previous year's score (Adrian).

4. In 2000, 7 of 17 SOI schools outperformed their matched comparison school on the state assessment for 3rd grade reading/literature.

5. In 2000, 8 of 17 comparison schools outperformed their matched SOI school on the state assessment for 3rd grade reading/literature.

Table 4.9 shows four years of state assessment data for grade 5 reading/literature. Points worthy of note in this table include:

1. For 2000, 11 of 17 SOI schools bettered their own previous year's performance; 8 of 17 comparison schools bettered their own previous year's performance.

2. Eleven of 17 SOI schools and 9 of 17 comparison schools showed a relatively stable or improving trend in 5th grade reading scores over the four years reported.

3. Of the 2 schools participating in the SOI Program for a third year (Adrian and Gray), both continued _to show consistent increases over the four years tabled.

4. In 2000, 6 of 17 SOI schools outperformed their matched comparison school on the state assessment for 5th grade reading/literature.

5. In 2000, 7 of 17 comparison schools outperformed their matched SOI school on the state assessment for 5th grade reading/literature.

Table 4.10 shows four years of state assessment data for grade 3 mathematics. Points worthy of note in this table include:

1. For 2000, 10 of 17 SOI schools bettered their own previous year's performance; 8 of 17 comparison

2. Ten of 17 SOI schools and a similar number of comparison schools showed a relatively stable or improving trend in 3th grade mathematics scores over the four years reported.

3. Of the 2 schools participating in the SOI Program for a third year (Adrian and Gray), both continued to show stable or slightly improving trends over the four years tabled.

4. In 2000, 8 of 17 SOI schools outperformed their matched comparison school on the state assessment for 3rd grade mathematics.

5. In 2000, 8 of 17 comparison schools outperformed their matched SOI school on the state assessment for 3rd grade mathematics.

Table 4.11 shows four years of state assessment data for grade 5 mathematics. Points worthy of note in this table include:

1. For 2000, 9 of 17 SOI schools bettered their own previous year's performance; 10 of 17 comparison schools bettered their own previous year's performance.

2. Ten of 17 SOI schools and a similar number of comparison schools showed a relatively stable or improving trend in 5th grade mathematics scores over the four years reported.

3. Of the 2 schools participating in the SOI Program for a third year, one showed a decrease from the previous year, which in its turn had been a dramatic increase over the previous two years (Adrian), and the other showed a steady increase over the four years tabled (Gray).

4. In 2000, 4 of 17 SOI schools outperformed their matched comparison school on the state assessment for 5th grade mathematics.

5. In 2000, 12 of 17 comparison schools outperformed their matched SOI school on the state assessment for 5th grade mathematics.

It has been previously noted that for the Year 3 evaluation, the state's assessment of students who had been third graders in 1997-1998 and were fifth graders in 1999-2000, provided a good opportunity to track the academic growth in key areas of a cohort of students who had experienced the SOI Program over two consecutive school years. In Oregon, such an analysis is possible because the scale used in statewide assessments is a continuous growth scale with numbers ranging from about 150 to 300, and each point on the scale is at an equal distance from the previous point on the scale, so that changes up or down can be charted and viewed as comparable over time.

Figures 4.1 and 4.2 show the growth in reading/literature and math of one cohort of Oregon elementary students over the two years that the SOI Program has been fully in place. Thus, for these figures, 1997-98 scores are considered pre -SO! and 1999-2000 scores are considered two years post-S01.

As seen in Figure 4.1, 5 of 17 SOI schools showed greater growth in reading/literature than their matched counterpart schools. On the other hand 8 of 17 comparison schools showed greater growth in reading/literature from 3rd to 5th grade than their matched SOI schools. Overall however, the average growth in reading/literature for this student cohort was almost identical for the two groups: 12.4 scale score points for SOI schools versus

12.9 scale score points for comparison schools.

Similarly, as seen in Figure 4.2, 7 of 17 SOI schools showed greater growth in mathematics than their matched counterpart schools. On the other hand 6 of 17 comparison schools showed greater growth in mathematics from 3th to 5th grade than their matched SOI schools. Again however, overall, the average growth in mathematics for this student cohort was almost identical for the two groups: 15.4 scale score points for SOI schools versus 15.7 scale score points for comparison schools.

6 2(

Page 50

Table 4.8: Four-Year Trends in 3"1 Grade Reading for 17 SOI Schools and their Matched Comparison Schools

1997 1998a 1999 2000 1997 1998 1999 2000

Statewide

Adrian*

209 209 210 213 b

CS1

209 209 210 213

208 208 209 210 217 211 208 215

Allen Dale

^{210 207 211 214}

CS2

^{209 210 211 215}

Bear Creek

^{209 212 212 215}

CS3

^{208 207 210 214}

Evergreen

^{209 209 210 208}

CS4

^{209 208 210 216}

Fairview

^{207 209 207 208}

CS5

^{205 203 209 205}

Fossil

^{206 nr 207 214}

CS6

^{208 206 208 213}

Goshen

^{209 211 211 214}

CS7

^{212 209 217 214}

Gray*

^{212 212 208 212}

CS8

^{211 210 209 212}

McGovern

^{209 210 210 211}

CS9

^{209 208 212 214}

Milner Crest

^{209 213 209 213}

CS10

^{212 209 210 210}

Rhododendron

^{211 209 213 210}

CS11

^{210 214 213 215}

Riddle

^{207 203 205 205}

CS12

^{205 208 207 214}

Stella Mayfield

^{209 205 207 214}

CS13

^{208 209 213 212}

Sweetbriar

^{210 210 210 219}

^CS14

^{211 209 209 214}

Thurston

^{210 208 209 217}

CS15

^{210 210 210 216}

Warrenton

^{211 213 211 206}

CS16

^{208 211 213 212}

Whitworth

^{205 200 203 205}

CS17

^{206 206 209 216}

Notes. *School participating in the pilot Program for three years: 'Statewide assessment

results prior to SOI Program; bAverage computed from the 34 SO! and comparison

schools (state average not currently available).

Table 4.9: Four-Year Trends in 5th Grade Reading for 17 SOI Schools and their Matched Comparison Schools

1997 1998a 1999 2000 1997 1998 1999 2000

Statewide

Adrian*

218 218 219 221b

CS1

218 218 219 221

216 217 224 229 209 219 218 220

Allen Dale

^{217 220 213 218}

CS2

^{219 219 222 224}

Bear Creek

^{216 218 219 222}

CS3

^{218 216 219 218}

Evergreen

^{220 222 221 222}

CS4

^{220 218 221 222}

Fairview

^{218 216 217 217}

CS5

^{215 219 217 219}

Fossil

^{213 216 225 219}

CS6

^{218 220 220 222}

Goshen

^{222 219 221 226}

CS7

^{224 220 225 224}

Gray*

^{217 219 220 224}

CS8

^{218 218 218 222}

McGovern

^{217 220 221 217}

CS9

^{215 215 219 218}

Milner Crest

^{221 223 216 226} CS10 ^{219 217 222 221}

Rhododendron

^{220 218 220 221} CS11 ^{218 218 221 219}

Riddle

^{216 218 217 216}

CS12

^{216 216 218 217}

Stella Mayfield

^{217 216 219 220}

CS13

^{223 218 224 236}

Sweetbriar

^{217 218 221 222}

CS14

^{219 218 222 221}

Thurston

^{217 217 219 221}

CS15

^{217 220 224 220}

Warrenton

^{217 218 216 221}

CS16

^{217 223 221 221}

Whitworth

^{213 216 219 216}

CS17

^{215 214 220 223}

Notes. *School narticinating in the pilot Program for three years: 'Statewide assessment results prior to SOI Program; °Average computed from the 34 SOI and comparison schools

(state average not currently available).

Page 52

Table 4.10: Four-Year Trends in

3rd

Grade Mathematics for 17 SOI Schools and their Matched Comparison Schools

1997 19988 1999 2000 1997 1998 1999 2000

Statewide

Adrian*

204 205 206 208b

CS1

204 205 206 208

201 201 204 206 205 202 206 202

Allen Dale

^{204 201 205 209}

CS2

^{202 203 206 206}

Bear Creek

^{207 208 209 210}

CS3

^{202 203 204 206}

Evergreen

^{205 203 206 203}

CS4

^{206 204 206 211}

Fairview

^{201 204 203 203}

CS5

^{203 203 203 200}

Fossil

^{198 nr 208 208}

CS6

^{V() 202 204 204}

Goshen

^{205 206 208 206}

CS7

^{207 209 214 210}

Gray*

^{205 205 205 207}

CS8

^{207 204 204 207}

McGovern

^{202 206 206 207}

CS9

^{203 207 208 210}

Milner Crest

^{203 202 204 208} CS 10 ^{205 208 206 206}

Rhododendron

^{208 206 208 206} CS 1 1 ^{207 208 208 210}

Riddle

^{205 198 199} 206

CS12

^{202 202 202 207}

Stella Mayfield

^{204 201 204 204}

CS13

^{202 208 207 207}

Sweetbriar

^{207 206 208 214}

CS14

^{206 203 207 207}

Thurston

^{205 205 206 212}

CS15

^{205 206 203 210}

Warrenton

^{207 208 210 203}

CS16

^{204 207 210 208}

Whitworth

^{201 200 203 204}

CS17

^{198 203 204 208}

Notes. *School participating in the pilot Program for three years: aStatewide assessment results prior to SOI Program; 'Average computed from the 34 SOI and comparison

schools (state average not currently available).

Table 4.11: Four-Year Trends in

5th

Grade Mathematics for 17 SO! Schools and their Matched Comparison Schools

1997 19988 1999 2000 1997 1998 1999 2000

Statewide

Adrian*

217 218 219 220b

CS1

217 218 219 220

215 215 228 224 211 217 213 217

Allen Dale

^{214 218 219 218}

CS2

^{219 218 219 219}

Bear Creek

^{215 218 220 222}

_CS3

^{217 215 216 217}

Evergreen

^{220 219 219 223}

CS4

^{218 219 221 221}

Fairview

^{216 219 215 215}

_CS5

^{215 218 217 218}

Fossil

^{213 215 220 213}

CS6

^{218 218 218 224}

Goshen

^{218 221 219 222}

CS7

^{222 221 229 225}

Gray*

^{216 216 218 221}

CS8

^{217 216 218 222}

McGovern

^{214 219 218} 217

CS9

^{213 213 216 217}

Milner Crest

^{216 219 213 222} CS10 ^{218 217 219 223}

Rhododendron

^{218 217 219 220} _CS11 ^{214 218 217 219}

Riddle

^{212 215 216 212}

CS12

^{214 217 121 219}

Stella Mayfield

^{216 219 220} 217

CS13

^{219 214 219 219}

Sweetbriar

^{217 218 218 220}

CS14

^{216 217 222 224}

Thurston

^{216 217 217 220}

CS15

^{219 221 223 211}

Warrenton

^{215 219 215 219}

CS16

^{218 221 224 222}

Whitworth

^{213 212 216 214}

CS17

^{215 214 218 221}

Notes. *School participating in the pilot Program for three years: aStatewide assessment results prior to SOI Program; °Average computed from the 34 SOI and comparison schools (state average not currently available).

Page 54

66

Statewide Adrian*

Allen Dale Bear Creek Evergreen Fairview Fossil Goshen Gray*

McGovern M liner Crest Rhododendron Riddle Stella Mayfield Sweet briar ThurSton W amen ton W hit worth

1998-2000 Growth in Readina

0 5 10 15 20 25 30

vl 12

A. 21

ist 1 1

114

.71

1

14

116

v. t 7

10

13 12

113

16

9

1..8

f113

10

1JA

0 SO I Schools Comparison Schools

Notes. *School participating in the pilot Program for three school years.

Figure 4.1: Growth in Reading from 1998-2000 for the Grade 3 to Grade 5 Cohort in 17

SOI Schools and their Matched Comparison Schools

Statewide Adrian*

Allen Dale Bear Creek Evergreen

Fairview Fossil Goshen Gray*

M cGovern

M ilner Crest Rhododendron Riddle Stella NI ayfield Sweetbriar Thurston W amen ton hitworth

0 5

1998-2000 Growth in Mathematic(

10 15 20 25 30

115

lb

17

117

Ill

1 1b

20

123

192

16 18

10/

11

li

I 14

14 10

14

I16

17

20

15 14

18

21

Et SO1 Schools o Comparison Schools

Notes. *School participating in the pilot Program for three school years.

Figure 4.2: Growth in Mathematics from 1998-2000 for the Grade 3 to Grade 5 Cohort in 17 SOI Schools and their Matched Comparison Schools

Page 56

68

Summary

In summary, these trend data indicate that there is little practical difference between SOI and comparison schools' academic performance at grades 3 and 5 in reading/literature and mathematics. Pair-wise graphical

analyses by school, grade, and subject showed little conclusive difference between SOI and comparison schools. Many schools in both groups improved in 1999-2000 over previous years' performances, and this is commendable.

It should be noted that one must exercise great caution in comparing any school's year-over-year performance because one is comparing the performances of different cohorts of children. Still, it is not unreasonable to expect that schools and students have become more familiar with Oregon's standards and state assessments, and therefore are more able and focused in terms of instruction and assessment around standards. Thus, although many schools in both groups improved, one must compare performances of school pairsto be able to judge the value added to Oregon elementary schools' academic performance by the SOI Program. In this regard, there is little to choose between comparison schools and SOI schools on statewide assessments over the four years examined.

Furthermore, this year of the program evaluation affords a unique opportunity; that is, to examine the growth in reading and math of a relatively intact group of elementary-aged students as they progress from grades 3 to 5, with grade 3 scores being "pre-treatment" and grade 5 scores being "post-treatment." This comparison, whether pair-wise or group-wise again showed that there is little if any difference, in the aggregate, between SOI schools and comparison schools in terms of the academic achievement measures analyzed.

Therefore, the answer to the question "What value does the SOI Program add to the academic performance of 17 Oregon elementary schools, over and above that of similar Oregon schools which do not utilize the Program?" must at this stage be, "No discernible value in terms of academic performance."