Rivkah Sass of the Omaha (NE) Public Library; and Vicki Terbovich of the Maricopa County (AZ) Library District. Many of these terms are also defined in the glossary at the end of the book. In the New Planning for Outcomes process, goals are developed from an assessment of community needs and are statements of ideal outcomes of selected library service responses.
In New Planning for Results, objectives are statements of how the library will measure its progress toward a goal. New Planning for Results emphasizes that libraries must base their activities on community needs. Meeting the needs of users then demonstrates the library's contribution to solving community problems.
SELECT A PROGRAM FOR OUTCOME MEASUREMENT
In outcome measurement, the library plans programs that will lead to outcomes for the users. Long-term outcomes are benefits to the user that do not become visible and measurable until a minimum of two years after the start of the program. Long-term outcomes answer the question: "What difference has our program made to the user in the long term?".
They are milestones in the life of the project, events that are necessary for long-term success. As outcome measurement focuses on ends rather than means, the adaptation is seen as a positive move by the library. If the users do not use the program frequently, the long-term results of the program cannot be achieved.
DETERMINE THE INTERIM AND LONG-RANGE OUTCOMES
As you saw in the last chapter, interim outcomes (sometimes called immediate or intermediate outcomes) occur sooner than long-term outcomes. Of course, a program that increases user satisfaction with the library (interim outcome) may have another interim outcome or a long-term outcome that is meaningful to the user. In this case, getting a job is the final so what, or long term outcome (sometimes called an end outcome).
Deciding where to set intermediate and long-term outcomes is difficult. Greater satisfaction of library users will be a long-term result of employee training. The library conducts a customer satisfaction survey every two years and will be able to determine whether the training was successful in terms of a long-term outcome based on the satisfaction ratings.
MAKE OUTCOMES MEASURABLE
A well-chosen pair of indicators and results describes a situation that makes the program's effect clear. Goals must be set locally, by individual programs, based on their knowledge of the circumstances and participants. Results: 20 percent (9) of participants who attend all programs in the series find a job within one year of completing the program.
80 percent (26) of the new immigrants in the citizenship class will double the number of independent visits to the library by the end of the semester. After completing this step, you should have a statement of results for each of the indicators for each of your results. The benefits of the data collection activities must outweigh the costs to both the program and the participants.
DESIGN THE DATA PLAN
If the data collection is part of the actual service, such as reading comprehension testing during a literacy session, the provider may actually be the data collector. Six data collection methods are the most commonly used in all types of evaluation, including outcome measurement. Interviews, or structured dialogues with an individual participant, are one of the most time-consuming and expensive data collection methods.
It is the cheapest method of data collection in terms of creating an instrument. Note that focus groups are not among the six recommended data collection methods for outcome measurement. The next step is to formulate how you will use this data collection method to gather evidence for the indicator.
At this point, you just need to decide which questions to add to the data collection tool. At the end of the data collection period, you will have dozens, hundreds, or even thousands of completed forms. If you have a small number of users and a simple data collection method (such as a short survey), your data analysis needs may also be small and simple.
This person is responsible for ensuring that data collection and analysis runs smoothly and according to plan. In smaller libraries, however, the roles of data manager and data collection supervisor can be combined. The data controller working with the data collection manager, if you have one, must monitor and track the data collection process.
It is important to develop a comprehensive schedule that includes the data collection needs at all stages of the process.
PREPARE FOR IMPLEMENTATION
What is the short-term benefit to the user as a result of the program/service. What the user says or does that reveals the achievement of an intermediate result. What is the long-term benefit to the user as a result of the program/service.
What does the user say or do that shows that the long-term outcome(s) have been achieved? Instead, ask everyone for input on developing the action plan (the final step in this task). If you need to find collectors, consider staff from other departments, members of the library's Friends group, or board members who can serve as data collectors.
They will administer both in-person and telephone interviews with users of the Job and Career Information Centre. These forms are the property of the library and must be submitted at the end of each shift. All individuals who will be involved in data collection or analysis should be part of the pilot test.
If you extract information from records, you can find out how much of the data is available or missing. To pilot test a survey or interview instrument, have two or three raters use the instruments with six to twelve participants who are representative of the people you will survey or interview. Ask them to keep track of the amount of time it takes them to complete the survey.
Conduct a similar data collection debriefing and ask participants if they were hesitant to be asked or interviewed.
MAKE THE MOST OF YOUR RESULTS
It is now seven months after the conclusion of the first cycle of senior internet classes. One indicator - a positive change in the student's self-assessment - was demonstrated by 90 percent of the teenagers. It is best to use each of the two types of questions: "provide" and "choose".
At the end of the workshop, you specified a new technique or behavior you wanted to try. If it is clear that the participants have no knowledge of the subject of the training, then a pre-test (survey or quiz) is not necessary. For example, if you are training staff to use a new automated system, you can assume that all the knowledge the participants have at the end of the training has been learned in the classroom.
Learning assessments accurately measure the amount of knowledge gained at the time of the test or survey. Only the ID number remains on the completed form, so few people know the identity of the respondent. However, if there are many errors or systematic problems, 100 percent of the data must be re-entered.
You will get the most complete evaluation of your program if you measure the results for all participants in the program. When measuring results, goals (also called deliverables) are directly related to the needs of the community and are always beyond what the library could achieve on its own. Impact is used as an example of Measure 2 in New Designing for Results: How well a service meets the needs of the people it serves.
Interpretation is the act of understanding data findings and drawing conclusions about them. It reflects an emphasis on the planning rather than the evaluation aspects of the approach. Note that in the design phase of New Planning for Results, the third measure is an output measure.
Program Suitability for Outcome Measurement
Program Suitability for Outcome Measurement (Cont.)
If–Then Exercise
If-Then Exercise (Cont.)
Outcome Selection
Outcome Selection (Cont.)
If "So what?" answer has not led you to an obvious end result, keep asking and answering that question. Remember that immediate results should be visible by the end of the first year of your program, and that intermediate results should be visible within three years. The long-range outcome may be the final outcome (very close to the community's goal) or it may be closer to an intermediate outcome.
Interim and Long-Range Outcomes
Interim and Long-Range Outcomes (Cont.)
Inputs, Outputs, and Outcomes
Inputs, Outputs, and Outcomes (Cont.)
Indicator Selection Criteria
Indicator Selection Criteria (Cont.)
If you answer "yes" to all three questions, you have completed the process of developing indicators for this outcome. If you answer "no" to any of the three questions, revise your final indicators again or develop one or more additional indicators for this outcome.
Outcome Statement or Objective
Selecting a Data Collection Method
Selecting a Data Collection Method (Cont.)
Relevance of Questions in a Data Collection Instrument
Data Plan
Data Plan (Cont.)
Outcome Measurement Plan
Outcome Measurement Plan (Cont.)
Users will see these long-term results at the end or after the program cycle:.
Action Plan
Type the individuals, organizations, or groups who will need or want information about the results of your program in column A. Determine what each individual, organization, or group listed in column A will want to know and enter it in column B. Identify the most effective communication channels for each individual, organization or group and enter them in column D.
Communicate Your Results
Program participation criteria Data collection method Data collection tool Data collection grid Data analysis types Data analysis software Data interpretation.
Use Your Results to Move Forward
