ACER Newsletter No. 39 July 1980

(1)

ACER Newsletter

No. 39 July 1980

Published by The Australian Council for Educational Research Limited

Radford House, Frederick Street, Hawthorn.

Victoria. 3122.

Edited by Peter Jeffery

Invitational Seminar on the Improvement of Measure- ment in Education and Psychology

One of the functions held this year to mark the 50th anniversary of the ACER was an invitational seminar which brought together experts who examined and discussed current developments in the field of measurement in education and psychology.

The seminar was held on Thursday 22 and Friday 23 May in the Council Chamber of the University of Mel- bourne . Sixty-one people attended, in- cluding participants from most parts of Australia, and from Canada, The People's Republic of China, Finland, Germany (FRG), Great Britain, New Zealand and the United States. A highlight of the occasion was the presence of Emeritus Professor R. L.

Thorndike, of Teachers College, Col- umbia University, who was especially invited by the ACER to give the open- ing paper. His visit to Australia and his tour of the eastern States was supported by the Australian American Educational Foundation.

The seminar was opened by the President of the ACER, Emeritus Pro- fessor P. H. Karmel, and the two days of the proceedings were chaired by Professor D. Spearritt of the Univer- sity of Sydney. An important part of the proceedings occurred at the dinner, which was held on the night of 22 May at University House, where Mr S.

S. Dunn, Chairman of the ERDC, gave

a lively and thought-provoking after-

dinner speech.

Objectives of the seminar

During the past two decades, there has been much effort extended by psychometricians in the development and perfection of latent trait measure-

ment models. Yet it is only within the last five years that measurement procedures based upon these models have begun to be used in the practice of educational and psychological measurement in Australia. A few practitioners in these disciplines have learnt of such procedures, but many still remain unacquainted with the features of the various latent trait models. Traditional procedures based on the true-score concept have many short-comings, such as the sample dependence of item parameters . In contrast, latent trait measurement models such as the Rasch Simple Logistic model are constructed so as to free item parameter estimates from this sample dependence, by specifying relationships between observable examinee performance on the test and the unobserved latent trait which is assumed to underlie performance on the test.

The nine papers presented at the seminar focused on both theoretical and practical aspects of latent trait measurement procedures. A brief summary is given for each paper.

Paper 1 Educational Measurement - Theory and Practice

Author Emeritus Professor R. L.

Thorndike (Teachers College, Colum- bia University, USA)

. This paper set the theme for the seminar. It described and contrasted important theoretical and practical issues concerned with psycho-

metric measurement and the analysis of test

data, that have arisen in education and psychology during the past 75 years. Thorndike pointed out that educational and psychological measurement during this time had involved parallel streams of practice and theory which had flowed together. Beginning with Spear- man's classic model of 'true-score' and error, he described the various models of test scores that had emerged. At present there are two ma- jor groups of competing models - the domain sampling (or criterion-referenced) and the latent trait models.

One of his conclusions was that latent trait models are useful in situations that focus on ability measures which reflect general growth, where the person's ability can be related to a vertical trait dimension, of increasing embodi- ment of the trait. The paper concluded with Thorndike's views on the directions that testing might take in the future.

Emeritus Professor R. L. Thorndike

Paper 2 Comparing Latent Trait with Classical Measurement Models in the Practice of Educational and Psychological Measurement

Author Professor J . A. Keats (University of Newcastle, Australia)

Professor Keats showed that latent trait measurement models had features, not pos- sessed by the classic true-score measurement models, which would lead to improved measurement in education and psychology. He compared the classic and latent trait models in the context of theories of cognitive development, and demonstrated the superiority of the latent trait approach, especially of the Rasch model, for modelling cognitive development.

The paper provided formulations of the one and two-parameter models of cognitive development within the Rasch measurement framework. These formulations are based on a principle which says that the form of the mathematical relationship between behav- ioural variables should be the same at the

(2)

group level as it is at the individual level. In addition, he related both ratio IQ and deviation IQ to these models.

Paper 3 The Use of Latent Trait Models in the Measurement of Cog- nitive Abilities and Skills

Author Dr. B. Choppin (National Foundation for Educational Research in England and Wales, Great Britain)

Choppin's paper was concerned with a 'wish to develop a sounder basis for the measurement of achievement'. He described the use of the Rasch model for item analysis, and gave three reasons for adopting this model:

(1) it is mathematically simple and convenient to use;

(2) the total correct score contains all the information about the person's ability;

(3) it predicts the behaviour of test items with considerable accuracy.

Choppin argued that more complex models, like the 3-parameter latent trait model, were difficult to apply because they required large samples, and did not have the estimation features of the Rasch model. In addition, he described and illustrated with examples how the Rasch model might be adapted to estimate person abilities and item difficulty parameters from incomplete response data matrices, using binary scoring, partial credit scoring, or scores provided by markers or judges.

Finally, he described current studies, with which he was associated, that employed Rasch measurement procedures, such as those deal- ing with the monitoring of school performance over time, and the development of item banks.

Paper 4 The Linear Logistic Test Model and Its Application in Educa- tional Research

Author Professor H. Spada (Uni- versity of Freiburg, Federal Republic of Germany)

According to Professor Spada, the Rasch model was deficient in at least two ways as it has been used. First, it did not allow for the explanation of item difficulties in cognitive psychological terms and, secondly, it was not geared to the assessment of changes in ability within an educational context. In his paper, he reviewed the developments leading to, and the assumptions underlying, the Linear Logistic Test Model (LL TM) which has been developed by several European researchers to overcome these deficiencies. This LL TM is a version of the Rasch Simple Logistic Model in which the item (difficulty) parameters are decomposed into linear combinations of more elementary parameters associated with cognitive operations (differing in complexity), and represent- ing, for example, effects of instructional measures. He also discussed the assumptions and features of the LLTM in the context of three different empirical studies which have applied this model. The first dealt with a model of thinking and intellectual development. The second was concerned with examining the effects of different instructional measures in the teaching of the instructional unit 'Nuclear power plants-dream or nightmare?' The third was concerned with the application of the LL TM to develop an instructional unit involving problems which require the recognition of func- tional relationships.

Paper 5 Using Latent Trait Mea- surement Models to Analyse Attitud- inal Data: A Synthesis of Viewpoints Author Dr. D. Andrich (University of Western Australia, Australia)

2

In this paper Andrich demonstrated how the Rasch latent trait model accommodated and synthesized both the Thurstone and Liker!

approaches to attitude measurement.

He described the Rasch Rating Model which is an elaboration of the Rasch Simple Logistic Test Model for ordered response categories for the cases of dichotomous and polychotomous response modes. In the model, an attitude statement is characterized by an affective (difficulty) parameter, with a parameter for person attitude. A feature of the model is the specifica- tion of a set of threshold parameters which designate, with respect to the underlying latent attitude continuum, the threshold or boundary points between response categories.

Furthermore Andrich examined the concepts of sufficiency, particularly in relation to the elimination of parameters prior to estimation, and the concepts of 'population-free' and 'sample-free'.

On the basis of empirical data and Rasch analyses, he suggested that the undecided category, typical in Likert-type attitude in- struments, was characterized by a different dimension from that of the other categories and should consequently be eliminated in final versions of an attitude questionnaire.

Paper 6 Conditional Inference in a Generic Rasch Model

Author Dr G. Douglas (University of Western Australia, Australia)

In this paper, Douglas gave an overview of the current state of the conditional inference argument in the context of Rasch latent trait models, and presented the generic form of a Rasch latent trait model.

Much of the paper was devoted to describ- ing and relating concepts which underpin Rasch measurement procedures-concepts such as specific objectivity, separability of parameters, additivity of parameters, sufficient statistics, and conditional versus unconditional maximum likelihood estimation.

Part of the paper dealt with the mechanics of the generic form of the Rasch Model, which was built on a general framework, and which allowed any number of facets determining the probability of success on a task to interact simultaneously: for example, 'one marker assessing the essay-writing ability of one sub- ject on one essay question represents a single individual interaction in a three-facet framework'. The generic Rasch Model included as special cases, among others, the Binary Item Analysis Model, the Andrich Rating Model, and the Rasch/Andrich Essay Grading Model.

Douglas also examined some numerical analysis problems associated with the conditional inference argument and gave some directions for future research.

Paper 7 The Use of Latent Trait Models in the Development and Analysis of Classroom Tests

Authors Dr J. F. Izard and Mr J.

White (ACER, Australia)

This paper described how teachers can develop a pool of calibrated items, using simplified test analysis techniques based on the Rasch model. Using the procedures enun- ciated by Wright and Stone (1979) in their recently published book, Best Test Design, the authors gave in the first half of the paper details for the design of progress and review tests based on the notion of a target population and the principle that the best test would be the uniform test in which items (difficulties) were located at equal intervals along the latent continuum. The examples discussed were of nar- row uniform tests, wherein all items had ap- proximately equal difficulties, as might be the case with mastery or criterion-referenced tests.

In the second half of the paper, the authors used procedures developed by Wright and Stone in conjunction with work sheets, to demonstrate how it was possible for 'teachers to use information from an item bank to con- struct tests with known characteristics, to check that their group of students performed on such tests in a manner consistent with the performance of the reference group used to set up the item bank, and to scale their own items to the continuum underlying the item bank'.

Paper 8 The Use of the Rasch La- tent Trait Measurement Model in the Equating of Scholastic Aptitude Tests Author Mr G. Morgan (ACER, Aust- ralia)

This paper presented the results of a study which assessed the capabilities of the Rasch Simple Logistic Model (RSLM) in the calibration and equating of final and trial forms of the Australian Scholastic Aptitude Test (ASAT).

Working within the existing framework of test development, various versions of the ASAT test were calibrated to a Rasch latent ability continuum associated with whole test and sub- tests determined mainly by item content.

Forms which were equated were linked by sets of common items from different content areas, at the whole test and sub-test levels.

The findings of the study indicated that it was possible to calibrate and equate at the whole test and sub-test levels. Moreover the results showed that the RSLM would attempt to fit to a common latent variable, groups of items that cohered in some fashion, such as through an underlying general scholastic aptitude factor.

In terms of the ASAT's current use, equated scores could be derived from different forms of the test and applied in the scaling process, thus bringing tests used in different years to a common scale.

Paper 9 Some Alternative Ap- proaches to the Improvement of Measurement in Education and Psychology: Fitting Latent Trait Models

Author Professor R. P. McDonald (Ontario Institute for Studies in Education, Canada; professor-elect, Macquarie University, New South Wales)

In the early part of his paper, McDonald considered the role of Spearman's linear common factor model in the interpretation of test scores and its relation to Lord and Novick's generic true-score model and Lazarfeld's latent linear model. He suggested: 'We can reasonably con- sider the normal ogive and logistic [latent trait]

models as nonlinear counterparts of the Spear- man model'.

Much of the paper was devoted to relating and contrasting McDonald's nonlinear (poly- nomial) factor analysis model with Rasch and Lord latent-trait models. In the paper he demonstrated that latent trait models, such as the normal ogive model, could be treated as special cases of nonlinear factor analysis by expressing the normal ogive curve as an in- finite series whose terms were polynomials that were mutually orthogonal under the assumption that the latent trait had a normal distribution.

Features of McDonald's model were that it allowed the analysis of covariance structures, and relied on the use of residual matrices for prescribed structure to assess model fit - a feature not available with the Rasch and Lord models.

continued on page 4

ACER Newsletter No. 39 July 1980

(3)

ACER RESEARCH MONOGRAPH NQ6

In this report we describe the for- tunes of 5000 17-year-olds at school, at work, and in between, dur- ing the summer of 1978-79. It is the first of a series in which we will report the findings of a three-year research program focusing on youth in transi- tion from school to work or to further education. The program is described in Chapter 1 of the Research Mono- graph and is funded by the Education Research and Development Commit- tee and by the ACER.

Three main questions

The study was designed to answer three main questions:

1 What influences achievement in the basic skills of literacy and numeracy?

2 How do these factors, along with achievement in basic skills, affect early school leaving?

3 Among early school leavers, what influences the 'success' of their transition from school to work, how well do they fare in the early years of their working life, and how do they see their future?

Patterns of influence

To answer these questions, we develop a model of the patterns of influence affecting achievement in school, school leaving, and workforce experience. This model is shown in the following figure. This particular arrangement of groups of influences has the following meanings: the three forms of school achievement in group 5 are influenced by all the factors in groups 1 to 4; school attainment (group 6) is influenced by groups 1 to 5; and occupational experience (group 7) is affected by all the influences in groups 1 to 6.

Thus we examine the following patterns of effects:

1 the effects of State/Territory of residence, school system (Govern- ment, Catholic , Independent), fam- ily rurality , school rurality, father 's occupation , father's education, mother 's education , family size, and respondent's sex (groups 1 to 4) on achievement in basic skills (group 5);

2 the effects of all of these factors, including achievement in basic skills (groups 1 to 5) , on whether or not an individual becomes an early school leaver and, among early school leavers, the number of year levels completed before leaving (group 6) ;

3 the effects of all of these factors (groups 1 to 6) on success in the early working life of early leavers and on their degree of involvement in further education (group 7)- that is , we examine influences on:

the time taken to find a job;

satisfaction with that job; the experience of unemployment; length of unemployment; prestige of present job and that of the job expected in 1984 ; any further education undertaken; and any further education planned in the future.

rn

^{1. State}

~

!

Data

During the summer of 1978-79, we mailed out questionnaires to some 6000 individuals aged between 17 and 18 and living all over Australia.

Eighty per cent of these were completed and returned to us. Four thou- sand nine hundred and nineteen Aust- ralians told us what it was like to be 17 years old at work, at school, and in between.

Analyses

We argue that complex social pro- cesses such as these cannot be described adequately with simple statistics. The statistics used may not be familiar to most readers and, for this reason , we take pains to explain their use and interpretation with several examples. The virtue of this approach is that it allows us to measure the effect of one factor upon another, other things equal. Thus, for example, it was possible to measure the effect of school system attended on achievement among students otherwise equal in terms of family background and the other factors mentioned above. We made 'other things equal' statistically to tease out the influence of each factor from the complex of influences that aftect youth in transition .

~ 5. School Achievement 2.School

~ ·System ^{I -} • Word Knowledge l Occupational Experience

I----,

• Time to first job

·Rurality · Literacy

·Numeracy · Satisfaction with first

f job

~ 3.Family · Ever unemployed

· Fathers occupation

.

· Months unemployed

· Fathers education 6. School Attainment · Current job status

· Mothers education ,_ _•Expected job status

· Family size Years of schooling I--< · Attempted further study

· Family rurality · Plans for further study

· Ethnicity completed

1-,

1

~

^4.Sex

~

Model of Patterns of Influence

Some illustrative findings

Some of the findings are presented under the following five headings: influences on achievement; stayers and leavers; the transition from school to work; the early career; the future. They are provided as excerpts from Chapter 6 of the Research Monograph. They should be accepted or rejected within the context of the com- plete report and not from this illustrative account alone.

Influences on achievement

One is left then with the likelihood that the State effects shown represent State differences in the practice and provision of education.

Other things equal, some ten per cent more Catholic school students and nine per cent more Independent school students achieve mastery of Literacy relative to students in Government schools .. : As with State effects and for the same reasons, we are unable t~

unequivocally attribute these differences to differences in what the three systems do and provide for their students.

Most likely rural students are disadvantaged by the isolation and limited provision they experience; however, this disadvantage seems to have little effect on the learning of basic skills.

A degree of disadvantage was clear, however, for students born outside Australia in a non-English-speaking nation.

Stayers and leavers

Other things equal and relative to NSW students some 22 per cent more ACT students stay on in school ... thirteen per cent fewer Tas. students stay on to the final years of high school.

3

(4)

:z<j)

Other things equal, nine per cent more non-

-J

masters of Numeracy become early leavers.

Students attending rural schools ... are a little less likely to stay on to the senior years of high school.

Relative to Australian-barns like them . some 19 per cent more [students born in a non- English-speaking nation] stay on to the final years of high school.

The fo llowing sections apply to early school leavers and not the age cohort generally.

The transition from school to work

Those at most risk during the period between school and work, in the sense of finding it harder to get a job, are those who leave school at the earliest opportunity. Finding a job seems to be a meritocratic process in wh-ich years of schooling is the criterion applied by employers ... Other things equal, each additional year of schooling reduces by one month the time it takes to find that first job.

The early career

With a couple of exceptions, early school leavers are at risk or not, according to their merits. Other things equal, the risk of unemployment is greatest for the least schooled and the least skilled ... While early leavers tend to be hired on their merits, they tend to be fired on their merits as well . employers evaluate the on-the-job performance of their employees, especially their numerical skills.

Other things equal it is the more numerate who keep their jobs longer and who end up in 'better' jobs ... Ten per cent fewer of those demonstrating mastery of numeracy have ever experienced unemployment, and on the average this group experiences a month less unemployment, and obtains higher status jobs than does the group who failed to master these basic numerical operations.

Each extra year of schooling ... decreases their probability of unemployment by fourteen per cent, decreases their overall unemployment by one and three-quarter months, and finds them jobs some 12 points higher in status.

Other things equal, early leavers from Catholic schools experience less unemployment though they experience no advantage in the time it takes to find a job ... Other things equal, 1 O per cent fewer early leavers from Catholic schools have experienced unemployment compared with those who attended Government schools . . early leavers from Catholic schools have been unemployed, on the average,· more than three-quarters of a month less than Government school early leavers .. . Catholic school early leavers end up in 'better' jobs . We infer that early leavers from these schools may have more of the traditional occupational virtues tha.t employers value.

The future

Failing to master basic Literacy and Numeracy skills increases the probability that early leavers will not go back to school to obtain more education ... Those who have done well at school ... plan to get more schooling and ex- pect to do well in the workforce ... the least capable and the least schooled of the early leavers seem willing to accept a less than bright occupational future rather than return to school.

Further education must be presented to youth as an experience different from that of traditional schooling and as one that provides

them with skill capital that can be traded on the

open market.□

4

Item Bank Meeting 26-27 May 1980

At least five groups of educators are benefiting from supplies of tested, classified, and retrievable items which are available through item- banking fac ilities. Teachers with ac- cess to appropriate measurement in- struments are able to improve their teaching programs through the diag- nosis of student strengths and weak- nesses. Curriculum developers pro- duce banks of items as an integral component of curriculum materials, and item bank procedures are used in curriculum evaluation. Educational researchers are able to obtain high quality achievement , ability, and at- titudinal measures which are valid for particular groups of learners. Ex- amination bodies responsible for the certification and moderation of individual achievement are assisted by item-banking practices, as are those who are responsible for the monitoring of educational achievement.

Representatives of these groups from all over Australia attended a meeting at the ACER on 26-27 May 1980. The meeting was chaired by Dr J. P. Keeves, Dr G. R. Maddocks (Assistant Director-General, Cur-

riculum and Planning, Victoria) opened the meeting, Professor H. Spada (University of Freiburg , Germany) and Dr B. Choppin (NFER) presented lead papers and discussion papers were presented by Mr D. Palmer (Tasman- ian Education Department), Dr

J.

F.

Izard (ACER), and Mr

J.

D. White (ACER). The meeting discussed the uses of item banks for assessment, examinations, and monitoring and considered the role of a national item banking consortium. The meeting felt that the ACER could act as a clearing house for materials, information, research findings , expertise, and developments in item banking in Australia and overseas. It was also felt that it may be desirable to set up a small prototype on which a feasibility study could be based. D

continued from page 2

He reported the results of computer calcula- tions with constructed examples. These results showed that his model for the analysis of covariance structure produced similar results to those of the normal ogive model. An advantage of the model he proposed was that it did not require prior examination of the dimen- sionality of the data, but a disadvantage was its assumption that the latent trait was normally distributed.

To make available the papers and deliberations of the seminar to a wider audience, the ACER will publish the proceedings in 1981 under the editorship of Professor Spearritt. D

Evaluation of the EPUY Program

The report prepared by Dr Graeme Hubbert earned praise and commen- dation from The Honourable W. Fife, MP, Minister for Education, in the House of Representatives in Can- berra on 21 May 1980. The report of the study showed that the EPUY scheme, for school leavers who found it hard to get or to keep jobs, made a high proportion of them 'more employable', in spite of the fact that many taking part in the scheme held negative attitudes towards for- mal schooling. The ev.aluation study focused on 72 EPUY courses in New South Wales, Queensland, South Australia and Victoria. These courses lasted between six and 20 weeks and concentrated on literacy and numeracy , guidance and counselling, and vocational and job-seeking skills. The study received support and assist- ance for large numbers of people associated with EPUY courses . Youth who had completed EPUY programs answered questionnaires as did unemployed youth seeking employment through Commonwealth Employment Service offices .

The evaluation study had two pur- poses: first to evaluate the effec- tiveness of EPUY in achieving the aim of improving employability, and secondly to study unemployed school leavers in order to learn more about their educational, vocational, and per- sonal needs. Using comparisons between the programs provided in different States, the report was able to make recommendations for the con- duct of future EPUY programs concerned with staffing arrangements, the selection of participants, and the content of the EPUY courses.

Copies of the report together with a summary are available from Austra- lian Government Publi shing Service bookshops. D

New from ACER

Books

School, Work, and Career: Seventeen- Year-Olds in Australia by Trevor Williams, Jeff Clancy, Margaret Batten and Sue Girling-Butcher (ACER Research Monograph No. 6) . $10.00 The Evaluation of Staff Development in Technical and Further Education: A Proposed Methodology by Adrian Ford- ham and John Ainley (ACER Research Monograph No. 7) . $10.00

Source guide

Serials in Education in Australian Libraries compiled by Margaret A.

Findlay and Elspeth Miller. Third edition available on COMfiche only $6.75