Some questions which could be asked of teachers during this initial implementation launch could include:
• Do the general and specific learning outcomes reflect the needs of learners in your class this semester?
• Were any of the outcomes entirely new to you? How have you been handling including these new aspects into your course?
• Were learners at the appropriate language proficiency level to meet the outcomes of the course?
• Were the outcomes and tasks described in the new curriculum appro- priate in terms of level?
• Do the course textbooks support the learning outcomes?
• Did the tasks you have been using to implement the new curriculum engage the learners? If not, why not?
• Was enough support provided to implement the new curriculum?
• What more support do you feel would assist you in this implemen- tation?
Curriculum planning in our context
the curriculum has had its greatest impact in ensuring students are able to enter mainstream academic programming with stronger English language proficiency in addition to preparation for the academic language milieu.
Trends in student achievement in mainstream courses appear to support this assertion. Even with the good will of the instructors and a relatively engaged and well-informed team, there were still areas of discomfort and dissonance that could only be overcome with consistent commitment to implementing the innovative vision represented in the plan.
Monitoring the Ongoing Curriculum as Lived
Monitoring the curriculum as lived is a more complex undertaking than simply overseeing the new curriculum launch. It is a process which is ongoing, but over a much longer period of time and must consider a wider range of data. As the ATESL Curriculum Framework explains,
“the curriculum-as-lived should contribute back into the curriculum as- planned. There must be a process in place for this to happen” (Chambers et al., p. S5-8). The Framework goes on to point out that if there is any kind of mismatch between the curriculum plan and the learners who are taking the course, it is the teachers who will first become aware of this and attempt to mediate such a gap. Teachers find themselves on the frontline in the classroom and over time they will adjust their teaching to meet their students’ needs, as they must. As teachers adjust their instruction and individuals are brought into update curriculum documents, adjust exams, and/or select new textbooks, “the curriculum-as-plan [is left] unattended or the revisions [are] haphazard as multiple individuals do what they can, or what they believe to be necessary, without the unifying logic of a shared vision or a shared understanding of learner needs” (Chambers et al., p.
S5-15). Some questions which could be considered by an organization when contemplating the ongoing monitoring include:
• Do we have adequate time and resources to ensure that a curriculum review yields fruitful information?
• Is the review possible with the resources at hand (e.g., qualified staff, learners who have fully experienced the course)?
• How much time and money are available to do the evaluation?
• Have there been changes in the external environment since the implementation?
• Have the requirements of the course’s stakeholders changed?
• If the review uncovers changes that need to be made, is there support for allocating resources to curriculum revision?
• How can we ensure all stakeholders can contribute to the process?
(Questions adapted from ATESL Curriculum Framework, 2011) In the next chapter, a model for approaching a meaningful evaluation of the curriculum as lived is presented whereby necessary adjustments can be made systematically to the plan, without the initial central essence of the plan being lost.
References
Aoki, T. T. (1993). Legitimating lived curriculum: Towards a curricular landscape of multiplicity.Journal of Curriculum and Supervision, 8(3), 255–268.
Bygate, M., Norris, J., & Van den Branden, K. (2013). Task-based language teaching. In C. Chapelle (Eds.),The encyclopedia of applied linguistics. Oxford, UK: Wiley.
Chambers, W., Gnida, S., Messaros, C., Ilott, W., & Dawson, K. (2011).ATESL adult ESL curriculum framework. ATESL, Canada.
Clandinin, D. J., & Connelly, F. M. (1992). Teacher as curriculum maker. In P.
W. Jackson (Ed.),Handbook of research on curriculum (pp. 363–401). New York: Macmillan.
Guardado, M., & Light, J. (2018). Innovation in EAP programmes: Shifting from teaching to learning in curriculum design. In L. T. Wong & W. L. Wong (Eds.),Teaching and learning English for academic purposes: Current research and practices (pp. 143–160). Hauppauge, NY: Nova Science Publishers.
Jamieson, J. S., & Papple, J. (2018). Academic Inquiry 2. Don Mills, ON:
Oxford University Press.
Lebauer, R. S. (2010).Learn to listen—Listen to learn 2: Academic listening and note-taking (3rd ed.). London, NY: Pearson Education.
Nation, I. S. P., & Macalister, J. (2010).Language curriculum design. New York, NY: Routledge.
Sauvé, V. (2009).A curriculum frameworks research project. Unpublished report for Alberta Employment and Immigration.
Stabback, P. (2014).What makes a quality curriculum? UNESCO International Bureau of Education, IBE/2016/WP/CD/02. Retrieved from https://
unesdoc.unesco.org/ark:/48223/pf0000243975.
Evaluating the Plan and Lived Experience of the Curriculum: Completing the Circle
Abstract Evaluating the Plan and Lived Experience of the Curriculum considers the final stage in the curriculum development process, evalua- tion. In fact, this chapter describes the ongoing cyclical nature of curricu- lum evaluation, rather than viewing it as one, final stage in a linear process.
The authors consider the practical aspects of curriculum evaluation, the why, when, how, and who of the process. Practical tools for the process are included, in particular, potential goals for the evaluation and questions to ask, and what data to gather to achieve these goals are discussed. As in every chapter, Guardado and Light provide a candid summary of how they engaged with this evaluation process in their EAP context. Finally, some resources to support curriculum planning are included.
Keywords Curriculum evaluation·Stakeholder consultation·Direct and indirect measures·Managing change
Questions to ask yourself about your context
• How do we evaluate change in our EAP program?
© The Author(s) 2020
M. Guardado and J. Light,Curriculum Development in English for Academic Purposes,
https://doi.org/10.1007/978-3-030-47468-3_6
103
• How do we measure the impact of the changes we make?
• Over time, how do we monitor changes in the curriculum-as-planned after it comes to life in the classroom?
• How do we determine if our curriculum plan has achieved its stated goals?
• Do we follow up to determine if our stakeholders are satisfied with the changes we have implemented
Why Evaluate?
Evaluating the Plan in Light of the Lived Experience
Throughout this book, we have strongly made the case for viewing the curriculum plan as only part of the process of effecting curricular change and determining the best outcomes for students. Our construct for this perspective has been largely informed by the writing of Ted Aoki, and Virginia Sauvé. Their work aligned with our own experiences as program administrators with program planning over the years; the notion that the perfect plan often looked very different in practice, if not immediately, then over time. The work of both Aoki (1993) and Sauvé (2009) under- lies this shift within a framework that elucidates the processes driving the shift from plan to lived experience. While the plan for a curriculum may be created in ideal conditions, by a small team, in reality, “curriculum is a shared event, it unfolds as instructor and learners interact with the
…course of study (curriculum plan) and with each other within the con- text of the [classroom]” (Chambers, Gnida, Messaros, Ilott, & Dawson, 2011, p. S5-13) (Fig.6.1).
In this book, we have endeavoured to consistently place curriculum evaluation as part of the circular process that includes planning, imple- menting, and evaluating a curriculum, in an ongoing process that is non- linear and perhaps can be viewed as never-ending. In this way, evaluating a curriculum in an EAP context will be a formative process. The goal is to inform ongoing improvement to the EAP program, and to continu- ously engage in a review of how students’ needs can be met to support their successful integration into mainstream undergraduate study within the realities of the context.
Some general inquiry questions which may guide the evaluation pro- cess may include:
Fig. 6.1 Model of EAP curriculum design (Adapted from Nation and Macalister [2010] by the authors, Guardado and Light [2018])
• How can this curriculum evaluation help us to identify strengths and weaknesses in our curriculum plan?
• Have our general learning outcomes created the desired impact on our EAP students’ language proficiency?
• Have we succeeded in identifying the best learning experiences to support our students’ transition to mainstream undergraduate study?
• Is our curriculum plan consistent with our pedagogical principles and preferred methodological approaches?
• How has the plan shifted as it has moved from a plan to a lived experience?
• Why has this shift happened and should these changes be incorpo- rated back into the plan?
When to Evaluate?
How Frequently Should an EAP Curriculum Be Evaluated?
Formative assessment of a curriculum plan should be undertaken fre- quently throughout the planning stages, the initial implementation, and within the first year of implementation. This type of monitoring ensures that the impact of the changes is being actively considered, that student outcomes are not jeopardized in the short run, and that teachers do not feel abandoned during a time of potential upheaval and change.
Undertaking a complete curriculum evaluation should be built into the initial curriculum plan and is likely to be completed on a longer time frame. ATESL Best Practices (2009) only recommends that each English as a second language program has a plan for regular curriculum review without stating any particular given time frame. Languages Canada (2018) suggests, in its standards document, that a curriculum review be undertaken every five years. This amount of time allows for full implemen- tation to have taken place, for real impacts of the curriculum change to be evident and measurable, and for the impact of the curriculum as lived to have exerted some influence on the plan’s interpretation. Realistically it allows for reallocation of resources to the project, with a reasonable dis- tance from the initial curriculum project. The purpose of the evaluation is not viewed as an opportunity to completely re-write the original docu- ment, but rather to keep the original document in a continuously adaptive process described by Diamond (2008) as the “transition from the ideal to the possible” (p. 127). A curriculum review can be a resource-heavy process, taking time and effort away from other seemingly more urgent endeavours. However, incorporating curriculum evaluation as a critical part of a curriculum project reframes it not as a nice to have option but rather as an integral element.
How to Evaluate?
In this section, we will consider how to undertake a curriculum evalua- tion: the processes, the questions, and the data which might be collected.
In each case, we will reflect on our own context and recommend resources that might be more applicable in a different context.
Direct and Indirect Measures of the Curriculum Impact When collecting data, it is widely regarded as good practice to collect both direct measures and indirect measures of the impact of a curriculum.
Beile (2008) succinctly defines these approaches:
Direct measures assess the skills, competencies, behaviors and attitudes fac- ulty expect the graduates to have attained. (n.p.)
Direct measures, then, may include such data as summative assessments, in-house or standardized, portfolios of students’ work, and class presen- tations. These direct measures may be the most effective way to mea- sure students’ learning against the specific learning outcomes stated in the original curriculum plan.
…Indirect measures of student learning rely on self-report data that ascer- tains the perceived extent of value of the learning experience. (n.p.)
Indirect measures, then, may include such data as surveys, retention and GPA data, focus groups, classroom observations, and exit interviews.
These indirect measures provide an enhanced understanding of the data collected using direct measures. They offer an opportunity to understand how the curriculum was experienced which can offer deeper insights into the data such as summative assessment scores.
Questions to Ask During a Curriculum Review
Earlier in the chapter, we identified some overall questions to guide the curriculum evaluation process. In this section, we suggest some specific questions which may be helpful in developing survey questions or focus groups discussion points. A useful resource for developing curriculum in a post-secondary setting is Designing and Assessing Courses and Curricula, by Robert Diamond (2008). Although not focused on EAP or language
courses in particular, it offers a number of case studies and resources for curriculum development. From his suggested list of questions to ask dur- ing a curriculum evaluation, we have adapted the following list to be suited for an EAP curriculum evaluation:
Is the curriculum meeting the EAP program’s learning goals?
• Is the curriculum, as planned and lived, consistent with the organi- zation’s vision and mission?
• Were the assumptions we made about the students’ language levels, motivations, and academic goals accurate?
• Are the general and specific articulated in our plan still appropri- ate for the students and having a positive impact on their academic goals?
• Has the language of our curriculum plan been easily understood by teachers, students, academic faculty partners, and other stakeholders?
• Are the learning outcomes still the most relevant for the language requirements of EAP graduates?
• Are there stakeholders in the external community who we now iden- tify as significant partners, but perhaps did not initially consult with?
• Have we addressed a wide enough range of learning outcomes in our original plan?
• Have we considered issues beyond language proficiency such as those related to academic socialization, integration with the broader uni- versity community, a positive student life experience? Is there a place for these in our EAP curriculum?
Has the curriculum contributed to higher levels of academic language proficiency among the students?
• Do the formative and summative assessments reflect the learning outcomes described in the curriculum plan?
• Can we triangulate direct and indirect measures to demonstrate whether or not students’ proficiency levels are matched by confi- dence in their capacity to perform in mainstream programming?
• Are the findings of summative assessments and ongoing GPA perfor- mance being adequately communicated to partners?
• Do teachers understand the impact of the curriculum on summative assessment data, and are they using this data to support improve- ments in student achievement?
• Do teachers have data which they have collected during class time to contribute to the curriculum evaluation?
Does the curriculum plan and lived experience reflect the pedagogical principles and preferred methodological processes proposed in the origi- nal plan?
• Is the curriculum sequence appropriate for the students’ academic language development according to the input of stakeholders and in alignment with the most recent literature?
• Has the curriculum been designed to ensure that each student has the opportunity to achieve their academic goals through improved language proficiency?
• Does the curriculum encourage active, student-centred learning and does the plan adequately incorporate engagement with the local community, opportunities to understand technology, and develop intercultural communication skills?
• What level of awareness do the students have about the curriculum?
Does the program need to do more to explore the learning out- comes with students at the outset of the course and throughout the summative assessment process?
From these guiding questions, more detailed questions may be prepared with detailed considerations for the particular context.
Who to Consult During the Evaluation?
Most curriculum guides, specifically for language programs and others for undergraduate courses, recommend consulting a broad range of stake- holders in the data collection in a curriculum review. Languages Canada (2018) recommends consultations take place with students, teachers, administrators, and recruiters. Diamond (2018) suggests the consultation group also includes university administration, other faculties, and even accreditation bodies. For our purposes of evaluating an EAP curriculum plan, we suggest the following groups for consultation:
• Students
• Teachers
• Program administrators
• External stakeholders (academic faculty partners, registrar’s offices).
The Most Important Stakeholders: Students
We began our discussion of a curriculum renewal process with an exam- ination of the shift from a teaching paradigm to a learning paradigm. In this shift, the centre of the process is the students; their needs, goals, motivations, aspirations, and the processes they undertake as learners. In turn, the most important part of a curriculum evaluation must be the students. Student data can be collected either by direct measures, such as summative assessment performance and quality of class presentations, or by indirect measures including interview or survey feedback, GPA in academic courses, and classroom observations. Students may express con- fusion or discomfort with their EAP program in a wide variety of ways:
through feedback, through appeals of final grades, through enrolment rates, or through low engagement levels. The purpose of the curriculum evaluation is to investigate and analyze the students’ reactions to the EAP program, and adjust to enrich their experience and success in achieving their goals.
Curriculum planning in our context
I. Responding to the curriculum as lived:summative assessment