List of Appendices
Chapter 1 Introduction
1.3 Present Study
As stated earlier the ISAT has been used for 20 years. However, until now, no study has been conducted to examine this test comprehensively, especially based on the Rasch
paradigm of measurement. It is considered critical to examine thoroughly an instrument that serves as such a high stakes test.
In addition, because the items used in this study were obtained from an item bank, it is necessary to examine the stability of the item parameters of the test with respect to their item bank values. Although in practice it is assumed that item parameters are invariant over time they may change over time or across different groups.
Another area examined is the predictive validity of the test. The ISAT, as described earlier, is used as a selection tool to enter higher education studies. Therefore, the extent to which the test predicts academic performance in higher education studies need to be studied. This can be considered as an effort to build a sound validity argument to support the intended use of the test according to the Standard for Educational and Psychological Testing set by the American Educational Research Association, the American Psychological Association, and the National Council on Educational Measurement (AERA, APA, & NCME, 1999).
Therefore, this study examines the validity of the test by examining its internal structure based on the Rasch model and the Rasch paradigm, the stability of the item bank parameters and its predictive validity.
For the predictive validity purpose, responses of the examinees on the ISAT and their academic performance in universities are needed. Although the data of the ISAT responses can be obtained from the CEA, academic performance data are available only from the universities. Thus, the predictive validity of the ISAT can be studied only with the cooperation of universities.
To provide comprehensive results, it is desirable that data are obtained from examinees from as many fields of study as possible, both at undergraduate and postgraduate levels, and with evidence of their academic performance in universities. Therefore, the
universities chosen were those that used the ISAT to select students for various programs of study, had academic performance records for at least one year, and were willing to supply such data.
Two years before this study started, that is in 2005, two universities, which will be referred to as A and B, used the ISAT to select students for undergraduate studies for almost all fields of study. In the same year, for postgraduate studies a third university, C, used the ISAT, to select students for postgraduate studies in all fields of study in that university.
However, only university A and university C were able to provide data for the academic performance of those who were tested in 2005. Although university C was able to provide data of students’ academic performance for postgraduate studies from all fields of study, university A which had undergraduate data, provided students’ academic performance from only two fields of study, namely Economics and Engineering.
In 2005 university A used the ISAT to select students for undergraduate studies in a special scheme (not SNMPTN). In this scheme students who were in the top ten in their class in their third year of high school could apply to take the test and the ISAT was the only test administered to the applicants. In contrast, in selection for postgraduate studies by university C, the ISAT was not the only admission test. Tests in specific areas were also used.
As indicated earlier, in a selection situation, the number of applicants is generally greater than the number who are admitted. In this study, although the number of applicants is known, it is not clear how many applicants were actually admitted. Also not known was the cut score of the ISAT or the role of the ISAT in the admission decisions and whether there were some criteria or considerations in the admission decision other than the admission test results.
The complexity of the selection situation and the difficulty in obtaining accurate information regarding the selection decisions in general was acknowledged by Gulliksen (1950) 60 years ago. He asserted that in practical situations what other variables in selection were involved and how much weight was given to these variables is generally not known. To overcome this situation he suggested making the most reasonable guesses based on the available data.
Information on the selection ratio and the role of the ISAT in admission decisions will help describe the distribution of scores of those admitted and will show the degree of homogeneity of the scores. Homogeneity of scores is relevant in studying predictive validity. For example, if the selection ratio is very small and the ISAT is the only selection criterion, then it is expected that the scores will be more homogeneous and that high predictive validity in terms of correlatiosn using those scores will not be observed.
Although the selection ratio and the role of the ISAT in admission are not known, the distribution of the ISAT scores of all applicants including those who had academic records (admitted group) and not (non-admitted group) were available. They are examined to show the degree of heterogeneity of the ISAT scores in the predictive validity sample.
The undergraduate and postgraduate groups may have different characteristics which may lead to different predictive validities. Therefore, examining predictive validity in these two groups is conducted although the available data show there is a considerable difference between postgraduate and undergraduate data in terms of the field of study and the number of students available.
Because different item sets were used for the undergraduate and postgraduate examinees, separate analyses have to be carried out for each group. Although they are
from the same item bank, the characteristics of the items in the sets may be different and the interaction with the persons may have an impact on the predictive validity of the test.
In summary, this study examines the internal consistency of the ISAT used in the selection of undergraduate and postgraduate students, the stability of the item parameters with respect their item bank values, and the predictive validity of the test.
Details of the aspects examined with regard to the internal consistency analysis are provided at the end of Chapter 2 and in Chapter 3. Because of the many aspects that are assessed in the internal consistency analysis, and to prevent redundancy, the results are reported in detail only for one set of data, in this case the postgraduate data. The results of the internal consistency analysis for the undergraduate data are reported as a summary. The reason that the postgraduate data were chosen to be reported in detail is that the postgraduate data were available earlier than the undergraduate data. The rationale and the procedure in examining all these aspects are presented in Chapter 3 and they are the same for both sets of data.