I EA Science Project
The IEA Science Project is a large- scale cross-national investigation of relationships between the science achievement of students and various home and school factors associated with the achievement. IEA (the Inter- national Association for the Evalua- tion of Educational Achievement) has released the first results from this project which show the patterns of science achievement across the
19 countries in the study.
( See the Panel on Page four for the list of IEA international reports) The IEA results show that the performance in science of Australian students is quite high when com- pared with students in other coun- tries. The Australian results were based on samples of students from government and non-government schools in all six states at two population levels-Population II ( 14-year-old secondary school stu- dents) and Population IV (students in the terminal year of the secondary school).
Some information about the samples was given in ACER News- letter No. 11 (December 1971).
STATE MEAN SCIENCE SCORES From the Australian data it is possible to obtain state mean scores, and to examine reasons for the ob- served differences.
The state mean values are given in Tables 1 and 2.
For each population the states
are shown in the same order: New South Wales, Victoria, Queensland, South Australia, Western Australia and Tasmania.
At the Population II level, Queens- land, South Australia and Western Australia form an upper group, Vic- toria and Tasmania form a lower group, and New South Wales is somewhere in the middle. For popu- lation IV South Australia is notice- ably higher than the others.
At the Population II level, differ- ences between the state mean science scores may be explained, in part, by differences in the amount of science done by the students.
TABLE/: Population II scores
State Science Amount of science score (hours/week)
NSW 24.4 2.98
Vic. 22.8 2.48
Qld. 26.8 3.61
SA 27 .1 3.84
WA 26.6 3.00
Tas. 22.9 2.85
Students were asked how many hours per week of instruction they received in science. State mean val- ues, calculated for this variable, are shown in Table 1. Since the data were derived from all students in the sample, and not only those cur- rently studying science, these mean scores take into account differences that may exist between states in the proportion of 14-year-old stu- dents studying science. Queensland and South Australia are the states
lruS
ACER
no 1 8, september 1973
with the highest mean scores on this variable, reflecting the emphasis placed on science in the secondary school curriculum by these states.
Victoria has the lowest rating, reflec- ting the relatively low emphasis placed on science, probably because the secondary school curriculum is less structured than in other states.
The number of hours per week spent on science instruction is about 50%
higher for the top group '( Queensl'and and South Australia) than in Vic- toria.
There is obviously a strong rela- tionship between the"icience scores and the ratings for 'amount of science' since the rank oqder on both variables is the same a'cross the 6 states.
I
One implication of• this relation- ship is that the mean science per- formance of a state could probably be increased, within limits; by in- creasing the mean number of hours of instruction per week.
The 'amount-of-science' measure used for Population II was not suit- able for Population IV. A 'subject- units' rating was calculated instead.
For each of the schools in the sample, information was available for all students at the Population IV level (not only the sample students) on the science subjects they were currently studying. Each student was given a 'subject-units' score which measured the amount of science he was doing. Students doing no science received a zero score. The 'subject-units' were summed for each school and divided by the num-
her of students to provide a mean 'subject-units' score for the school.
The school scores were averaged to provide a state score.
As for Population II, the Popula- tion IV 'amount-of-science' values take account of the proportion of students in the state who were studying science. The state scores on this 'amount-of-science' variable are included in Table 2.
TABLE 2: Population IV scores
State Science Amount of science score (subject-units)
NS-W 23.8 1.08
Vic. 24.2 0.97
Qld. 25.5 1.25
SA 28.1 1.42
WA 24.8 1.35
Tas. 25.4 1.24
South Australia has the highest rating. As for Population II it is about 50% higher than Victoria's rating. In other words, the Victorian students are studying an average of one science subject-unit each, while each of the South Australian students is studying about 1
½
science .subject-units.
There ls a rather strong relation- ship between the science mean scores and the amount of science ratings.
Another aspect of the IEA analy-
ses is to identify important factors in the school teaching-learning situ- ation that account for differences between the science scores of the students.
SCHOOL LEARNING CONDITIONS Many school learning conditions factors are confounded with student background factors, such as the sex and home circumstances of the students and the type of school they attend.
By means of stepwise multiple regression analysis procedures it was possible to identify school learning conditions factors that make an important contribution to student science achievement after making a statistical allowance for the back- ground factors.
The strongest factors are those associated with the amount of science done by the student. The tests were given to all students at the two population levels, not only students currently studying· science.
At both levels the analyses show that students currently studying science obtained higher science scores. Stu- dents receiving more hours per week of science instruction, and students doing more hours per week home- work also obtained higher scores.
Towards Alternative
Learning: Beyond the School Base
This is the third of a series of four articles by Ron Fitzgerald who in 1972 undertook an overseas tour to study and to examine alternative ways of educating adolescents.
The seventies have seen the set- ting up of new types of schools in make-shift buildings. In some over- seas countries, as well as in Aus- tralia, the street corner school now operates in rented halls. These innovations represent an attempt to establish new structures of relation- ships between teachers and students on one hand and teachers and parents on the other.
These more radical approaches seek to utilize community resources and to return some responsibility to
2
the community for the education of the young. Students make use of facilities-such as libraries, sports grounds and art galleries-already available in the community. These schemes have been most numerous in North America, but there-as in Australia-they remain still at the experimental stage and involve small numbers of adolescents. It remains to be seen whether they are viable on a larger scale.
Recent experience in America indicates a very high failure rate
·for ventures of this kind. The lack of conventional routines and rituals built into the conventional school has meant a constant concern with the need to define operational boun-
At both population levels students obtain high science scores at schools where the science teachers state that they make a considerable effort to base students learning of science on practical work. At the 14-year-old level the number of hours per week spent by the science teachers at the student's school on preparation of lessons, marking of tests, and reading to keep up with science subject matter was linked to high student science achievement.
At the pre-tertiary level, high science scores were linked to stu- dents who considered that there was a liberal approach to discipline in their schools.
High scores were also linked to schools which had laboratory assist- ants and other ancillary staff, and to schools where decisions about the syllabus and teaching methods of science courses were not left to in- dividual teachers but were made by the head of the science department of the school or external bodies such as an education department.
Taken together, these and other IEA analyses indicate that effective learning of science takes place in a consistent school environment where students receive competent systematic instruction in carefully structured science courses.
daries. Also the difficulty of finding the right balance between structure and freedom has often led to a tota I abdication of the teacher's role and authority.
The heavy physical demands made on teachers involved in the street corner schools also help to explain their instability. There are the routine administrative problems of having students scattered in dif- ferent parts of the community at the same time and the enormous work-load of teachers due to much personal interaction with adol- escents. The unusual stress on such aspects as autonomy and self- development also create special difficulties for teachers concerned with punctuality and co-operation on the part of students.
The future of the street corner school as a general alternative to the conventional school seems de- pendent on the development of
ACER Newsletter No. 18: September 1973
sound strategies of management and control linked with a clear cut rationale for their learning pro- grams. So far these key elements have been lacking. The current search for alternatives appears due more to cultural change rather than to new pedagogic theory. Should this situation remain, then the present interest in innovation with its various manifestations appears certain to be little more than a passing phase. If so, the hard-won experience of educators in these new and complex areas of social action wi 11 have been largely lost.
There is therefore a need for those involved in these experiments to analyse and document both the less successful as well as the more effective aspects of their pioneer work.
BANK EM
PROJECT
THE GERM OF AN IDEA In October 1972, the Australian Council for Educational Research held an invitational conference, on 'ACER 's Priorities and Program on Testing and Evaluation', at which representatives of all Australian de- partments of Education and exam- ination boards were present. On the recommendation of that conference, the ACER agreed to explore the pos- sibility of producing item banks in several subjects as 'a flexible means of providing test material for a var- iety of uses by class teachers and those concerned with comparability or moderation across and within schools'.
One positive feature of such a project, which was emphasized by the participants at the conference, was that professional expertise could be brought into testing with- out the limitations often present in internal and external examining.
In essence, an item bank in a particular subject area is a collection of test questions in that area, avail- able for use by teachers or exam- ACER Newsletter No. 18: September 1973
iners. However, recent literature has stressed the value of central item banks in providing teachers interes- ted in measuring educational ach- ievement with test questions, which are at least as professionally accept- able as the questions used in public external examinations. At the same time teachers would be freed of the restrictions imposed by the use of external examining procedures.
POTENTIAL USERS
The principal potential users of item bank material in Australia in the near future are:
1. State education departments or examining boards interested in obtaining common data for mod- erating or adjusting internal school assessments.
2. Education authorities (e.g. state education departments) interes- ted in monitoring the achieve- ment of specific and specified educationa I objectives across a state or region.
3. Teachers involved in classroom assessment of student perform- ance for either grading or diag- nostic purposes.
In some states, the first two needs are seen as of immediate signifi- cance. Within a few years, at the 10th grade level of schooling which is to be the focus of the project, the third use will become of greater concern.
DEVELOPMENT PLAN Items in three subjects - mathe- matics, science, and social science - at the grade 10 level, have al- ready been written and are at present being edited in readiness for trial testing in schools in 1974.
The preparation of an item bank in each one of the three chosen sub- jects will proceed broadly along the following lines:
1. A classification of the relevant cognitive objectives, arranged in a grid so that each objective can be readily 'tagged' by a letter and number for ready access and identification. This process in- volves decisions on the degree of detail by which the various objectives of a subject ou@ht tb be specified.
2. Several workshops involving 10-20 teachers for four days to write test questions based on the prepared grid of objectives have already been held and more are planned. It is intended to have at least three such work- shops for each subject.
3. Items prepared during the work- shops are being edited and sup- plemented at ACER, with the assistance of teachers seconded briefly from secondary schools in the various state teaching services.
4. The edited items will be trial tested on a sample of appro- priate students in each state.
This testing will involve the co- operation of state education departments and classroom teachers.
5. Item analysis and the preparation of item ·data will be carried out at ACER and the actual set of items forming the item bank will be prepared for publication or for whatever use is decided by that stage.
USING THE ITEM BANK The preparation of test questions on a professional basis allows for certain essential data to be collected and continuously updated. Such data include approximate difficulties of the questions for groups the content objectives are concerned with, and in the case of multiple-choice ques- tions, the way the distractors are functioning.
An indication of one possible use of an item bank for educational measurement purposes may be ob- tained from the monograph on Item Banking by R. Wood and L. S. Skur- nik ( NFER, Slough, Bucks, 1969).
In each of the proposed subject areas it is planned to have two paral- lel but independent collections of test items. One collection will be available on a secure basis to any state-wide authority (e.g. an exam- ination board) for use in moderating or monitoring or reference test pro- jects. It is expected that the items will form the basis of centrally pre- pared tests which will be used to gather statistical data needed to ai':hJ.ise or instruct schools on the allocation of grades within a class 3
or school. The techniques of using tests constructed from banks of items for-such purposes are complex, and it is expected that each auth- ority will determine for itself how performance on such tests will be used to allocate or suggest the pro- portion of various grades or marks to make the awards comparable from school to school.
The ACER proposal is to provide test items of professional standard, each item being classified on the basis of the content objective it serves and broadly on the basis of the process (e.g. recall) required to answer the item. A letter and number grid will be used for ready identifi- cation of each objective tested. The user would be provided with details of this grid. To simplify usage, it should be possible to provide
(a) a list of identification numbers of all items in the bank. In addi- tion to the letter and number, which specify the cell on the grid, this would also provide a rough idea of difficulty ( on for example a five-point scale), and a symbol indicating the overall appearance of the item ( e.g. verbal, tabular, numerical, etc); and
(b) an alphabetical index to the con- tent list, perhaps in rather more detail, which directs the user to the appropriate identification number or numbers.
For each item it is intended to supply data, based on thorough pretesting, indicating the approxi- mate difficulty of the question, and discrimination indices for the cor- rect answer and all distractors. It is recognized that item difficulty could be a function of either the central or peripheral role the ob- jective being tested plays in a par- ticular course; and it is conceivable that two sets of item characteristics might be necessary.
The second parallel item bank is intended for direct use by teachers.
Several forms of access by teachers to such a bank of items are being explored. With a fair availability of computer facilities the following is tentatively suggested as one pos- sible arrangement:
A teacher might contact a test ser- vice requesting, for example, copies 4
of a test consisting of 50 items
'covering 35 'cells' of objectives.
One such objective in mathematics might be 'recall of the rules for indices'. The test service facility would print a mastE;)r for the teacher to duplicate with, the actual test items being chosen from the bank by a sampling process. Answer sheets could be returned to the test service agency to be processed, using optical reading and computing facilities; the teacher would receive a print-out giving details of his students' performance on the test and on each item, such performance being compared with the perform- ance of students, already entered in the data bank, on each item. In turn the new data would be used to modify the data bank.
It is felt at present that too ready an access to the bank of items with- out guarantee of feed-back informa- tion might lead to misuse and to the wastage of much of the initial effort in constructing the items and gather- ing the initial data. If for example the actual items are available to teachers to draw on directly for preparing class tests, the tendency could be to select items which would maximize performance at the expense of the assessment of the general objectives. The procedure suggested above has the merit of leaving the choice of the actual set of items appearing in a test to a process of random sampling while allowing teachers full control over the specific objectives being tested. It is also likely that if as expected the need for state-wide compara- bility of gradings and assessments declines, this second parallel bank will become the greater value and will be reinforced by test items from the original 'secure' bank.
An alternative proposal for use of a non-secure item bank would involve classroom teachers having a greater say in the composition of the tests.
A later stage of development could be the provision of suggested remedial material related to the items in the bank, should these serve primarily a diagnostic function.
It is envisaged that a continuous check on the effectiveness and use of each item in the bank wquld be
carried out and that provision would be made to eliminate and add items as required.
The above proposals are tentative and, although item writing work- shops have already been held, no other aspect of this project is fina I ized. It is hoped that further discussion will result in moulding a firmer shape for the project.
The ACER Newsletter is pub- 1 ished quarterly by the Austra- lian Council for Educational Research, Frederick Street, Hawthorn, Victoria 3122. Com- munications should be ad- dressed to the Editor at this address.
THE INTERNATIONAL IEA REPORTS
The IEA Science Project is only one part of the current set of IEA sur- veys. the international results of these surveys in 6 subject areas will be published in a series of 9 vol- umes-the International Studies in Evaluation. All the volumes will be published jointly by Almqvist & Wik- sell, Stockholm, and John Wiley, New York.
I Science Education in Nineteen Countries
( L.C. Comber and J.P. Keeves) II Literature Education in Ten
Countries (A.C. Purves)
Ill Reading Comprehension Educa- tion in Fifteen Countries (R. L. Thorndike)
IV English as a Foreign Language in Ten Countries (E.G. Lewis) V French as a Foreign Language in
Seven Countries (J.B. Carroll) VI Civic Education in Ten Countries
{R.F. Farnen et al.)
VII The National Case Study: An Empirical Comparative Study of Twenty-one Educational Systems
(A.H. Passow et al.)
VIII An Empirical Study of Education in Twenty-one Countries: A Technical Report (G.F. Peaker) IX The !EA Six Subject Survey: An
Empirical Study of Education in Twenty-one Countries
(D.A. Walker)
The first 3 volumes were published in May 1973. The other 6 volumes will be published in 1974.
Australian Results
The first book to report Australian results from the IEA Science Project was published in May 1973: Rosier, M .J. Science Achievement in A us- tralian Secondary Schools. Hawthorn: ACER, 1973.
ACER Newsletter No. 18: September 1973
