Before developing an assessment instrument, you must first determine why you are conducting the research. What is the purpose of the survey? The an- swer to that question will guide you through the development process.
For example, a short information literacy pre- and posttest to address short-term skills deficiencies or to determine current skill sets may be suffi- cient when you are assessing the following:
Students’ acquisition of individual skills; e.g., developing a thesis statement or search strategies
Students’ knowledge of local resources and facilities; e.g., what is the name of the library’s online catalog? Where is the reference department located?
Subject-specific knowledge for a particular course; e.g., what does ERICstand for? Where would you go to find scholarly informa- tion on child abuse and domestic violence?
Other information you may want to know; e.g., attitudes about or knowledge of librarians
As a rule of thumb, pre- and posttests should not be identical, though they may be similar. Posttest results may be biased if students memorize the correct
Developing Assessment Instruments 155
response. If tests must be identical, queries should be renumbered at the very least.
On other occasions, a pre- or posttest would not be suitable. If, for ex- ample, the purpose of the assessment is to determine baseline knowledge for a large number of students representing a diversity of majors—either once or over a period of time (longitudinally)—then a more comprehensive and gen- eralized instrument is in order. The goal of these types of surveys is to assess students’ general standards and skills while de-emphasizing their knowledge of subject-specific and localized resources.
The important point is to think through your purpose before developing the assessment instrument. The goals and objectives of the assessment—
along with any research questions and hypotheses—should be directly con- nected to and, ideally, developed prior to the development of the actual survey queries. It is far better to have the questions in mind before devel- oping the survey than to try and use survey data to answer research questions developed after the survey has been completed.
In the case of the UMBC’s information literacy program, the following goals and objectives were developed for assessment purposes:
To gather baseline data on the information literacy skills of UMBC students
To use the data gathered to assist in the development of an infor- mation literacy program for students, faculty, and staff at UMBC
Although there were only two goals, they were interconnected and were crit- ically important to the future of the overall UMBC instruction program: the data gathered to determine baseline levels of information literacy skills were used to develop an information literacy program for the campus.
Other goals and objectives for assessment may focus on individual stan- dards, performance indicators, and objectives, such as the following:
To gather data on the information literacy skills of students enrolled in a particular course, at a particular academic level (under- graduate, graduate), etc.
To determine whether college-level students know when to recog- nize a need for information
To determine whether college-level students know how to access information effectively and efficiently
156 Developing Assessment Instruments
To determine whether college-level students know how to evaluate information and its sources critically
To determine whether college-level students know how to incor- porate selected information into their knowledge base and value system
To determine whether college-level students know how to use information effectively to accomplish a specific purpose To determine whether college-level students understand many of
the economic, legal, and social issues surrounding the use of information
To determine whether college-level students use information ethically and legally
Once you have developed specific goals and objectives for your survey, the next step is to develop specific research questions and hypotheses. What theories and assumptions do you have about your population? What do you know about possible results? What do you want to know, prove, or disprove?
Based on anecdotal evidence available in every library—primarily consisting of reference desk encounters, bibliographic instruction interactions, and the inevitable one-shot lectures—you should be able to determine what you know about the possible research results, what the important issues are, and what information would help you support or disprove your hypotheses.
The following research questions were developed by the UMBC Task Force to address the perceived deficiencies and needs of students at the uni- versity. These can also be used as a starting point when developing your own research questions and hypotheses.
What are the attitudes of college-level students about Christina S.
Doyle’s information literacy skills?3
To what extent do [UMBC] students tend to overestimate their information-literacy confidence levels?
To what extent are [UMBC] students unable to identify the basic elements of a bibliographic citation?
To what extent are [UMBC] students aware of what constitutes plagiarism?
To what extent are [UMBC] students familiar with the concept of fair use?
Developing Assessment Instruments 157
To what extent are [UMBC] students familiar with the concept of copyright?
Do [UMBC] students who self-report a high level of confidence with information literacy skills perform well when responding to questions that represent those skills?
Do [UMBC] students who self-report a high level of confidence with computers perform well when responding to questions that represent information literacy skills?
To what extent do [UMBC] faculty model good library use behavior?
To what extent do [UMBC] faculty encourage students to use the library?