When Developing Materials
Most course materials are developed to match a predetermined syllabus but very few are developed in accordance with a predetermined set of principled criteria. In our view this is why some of them achieve coverage of language items and language skills but are less successful in facilitating language acquisition and development. From our experi- ence of developing materials to match criteria we would recommend to all materials writers (whether they are developing materials for their own class, for an institution, for a Ministry of Education or for a commercial publisher, and whether they are devel- oping print or digital materials) that they make use of the set of procedures outlined below.
1. Establish a team(even if there are only two of you) so that beliefs can be shared and debated.
2. Brainstorm your beliefsby first of all individually listing at random what you believe best facilitates language acquisition and development and then by comparing the lists and producing a new list of commonly held beliefs. At the second stage we would recommend writers trying to justify their beliefs which are not commonly held by the team before the final list is compiled. Here is a small sample taken from our long random list of commonly held beliefs:
We believe that language acquisition and development is best facilitated by:
r
exposing learners to the target language in use;r
ensuring that the learners’ exposure is rich in quantity, quality and variety;r
ensuring that the learners’ exposure is meaningful to the learners;r
ensuring that the learners’ exposure is related to their lives, interests, needs and wants;r
ensuring that the learners’ exposure is contextualized;r
ensuring that the learners’ exposure is comprehensible to the learners;r
engaging the learners affectively;r
engaging the learners cognitively;r
providing the learners with opportunities to use the target language for mean- ingful communication;r
providing the learners with opportunities to make discoveries about how the target language is used.3. Categorize your beliefsby grouping them under headings. Here is a small sample of our category headings:
r
exposure to written language;r
exposure to spoken language;r
written communication;r
spoken communication;r
language content;r
topic content;r
skills content;r
motivation.3 Materials Evaluation
4. Convert your beliefs into universal criteriaby turning them into questions that attempt to predict the likely effects of the materials on their users. When doing this make sure that each question:
r
is answerable;r
only has one answer;r
is evaluative and not analytical;r
is valid;r
is useful.Here is a small sample of our universal criteria:
Reading Texts
To what extent are the texts likely to:
r
Expose the learners to language in authentic use?r
Expose the learners to language in typical use?r
Expose the learners to language which is meaningful to them?r
Expose the learners to language which is comprehensible to them?r
Expose the learners to language which relevant to their lives?r
Expose the learners to language which is relevant to their needs?r
Expose the learners to language which is relevant to their wants?r
Expose the learners to language which is recycled at intervals?r
Interest the learners?r
Engage the learners affectively?r
Engage the learners cognitively?r
Motivate the learners to read outside the course?Note that:
r
We would say that all 12 of these criteria are important for the selection or devel- opment of reading texts for any materials development. Other criteria could be added for larger projects such as the development of a global coursebook (for example, to what extent are the texts likely to have immediate appeal to a wide variety of learners?).r
A similar set of criteria could be developed in relation to the predicted effects on teachers in the target context of learning and another could be developed in relation to administrators.5. Develop a set of local criteria by first creating a profile of the learning context of the target learners and then deriving criteria from it. In the profile, information should be provided, for example, about the age, gender, language level, motivation, reasons for learning the language, needs, wants and interests of the learners, the qualifications, experience, age, preferred teaching styles, and workload of the teach- ers, and the duration, intensity and targets of the course(s). This is obviously eas- ier to do when writing for your own class or institution than it is for a national and (especially) for a global coursebook. To provide an informative profile for the latter types of books, research, observational visits and questionnaires are necessary and flexibility becomes the most important criterion. For example, when developing a national coursebook for Namibia (On Target, 1994), questionnaires were adminis- tered to students and to teachers throughout the country and very useful informa- tion was gathered (especially about the students’ preference for narrative texts and for serious and provocative topics such as drug abuse and domestic violence). Here
Materials Development for Language Learning
are some examples of local criteria for 15-year-old students at state high schools in Indonesia:
Listening Texts
To what extent is the topic content of the listening texts likely to:
r
Be acceptable in a Muslim community?r
Achieve sufficient topic familiarity for 15-year-old students in Indonesia?r
Be relevant to the lives of 15-year-old students in Indonesia?r
Stimulate affective responses from 15-year-old students in Indonesia?r
Stimulate 15-year-old students in Indonesia to think?r
Stimulate 15-year-old students in Indonesia to communicate their views?r
Appeal to both male and female 15-year-old students in Indonesia?r
Appeal to students in both rural and urban areas of Indonesia?r
Be of educational value to 15-year-old students in Indonesia?r
Motivate 15-year-old students in Indonesia to look out for English outside their classroom?It might seem strange to develop universal criteria before local criteria (i.e. criteria for any learner anywhere before criteria for the target learning context) but we have found that focusing on local criteria first can lead to a neglect of vitally important principles of language acquisition and can lead to the development of materials that appear to be locally relevant but which do not actually help learners to acquire and use the target language.
6. Develop a set of medium specific criteriaby first listing important aspects of the medium being used and then deriving criteria from them. For example, for paper materials you would list aspects of design, layout, instructions and illustrations (see Chapters 13 and 14). These aspects could also be important for digital materials but you would also need to focus on aspects of individualization, navigation, and interac- tivity (see Chapter 14). Here are some examples of medium specific criteria for paper materials in Indonesia:
Illustrations
To what extent are the illustrations likely to:
r
Help the students to understand the language connected to the illustrations?r
Help the students to use the target language for communication?r
Help the students to do activities connected to the illustrations?r
Be acceptable in a Muslim community?r
Achieve sufficient familiarity for 15-year-old students in Indonesia?r
Be relevant to the lives of 15-year-old students in Indonesia?r
Stimulate affective responses from 15-year-old students in Indonesia?r
Stimulate 15-year-old students in Indonesia to think?r
Appeal to both male and female 15-year-old students in Indonesia?r
Appeal to students in both rural and urban areas in Indonesia?Note that, as with many other categories and subcategories, it would be useful to ask analysis questions before doing an evaluation. For example, are the illustrations intended to be functional or just decorative?
7. Combine the three sets of criteriain a way which will make it easy to use them. We would advise the use of a table for each category and subcategory with the following headings:
3 Materials Evaluation
Category – Sub-category –
Criterion Gradea Commentb
aWe would recommend using a five-point scale with a criterion achieving 4 or 5 in order for it to be considered as successfully achieved.
bThe comment column can be used to record reasons and examples justifying the grade as well as suggestions for modifications that could lead to a higher grade.
When you have made use of the above set of procedures you will have a set of princi- pled criteria that you can use both to drive the development of your materials and (with some additions and modifications) to evaluate them during and after their development.
In this chapter we have focused on developing criteria prior to starting to write your materials. In Chapter 4 we will be focusing on evaluating materials prior to adapting them and in Chapter 5 we will be focusing on principled ways of actually developing materials.
When Reviewing Materials
If you are reviewing your team’s materials during their development it is important that you make use of the criteria you developed prior to starting to write them. It is also important that you review the materials that you have written and that other members of the team review them too. For example, whenOn Target(1994) was being written by a team of 30 writers in Namibia, each mini-team of three wrote a first unit. Each mini- team then reviewed all the other mini-team’s units and the best one (i.e. the one consid- ered by the most mini-teams to be the most likely to be effective with the target users) was determined and then modified in relation to feedback from the mini-teams. Each mini-team was then made responsible for writing a unit and for periodically review- ing another mini-team’s unit. Use was made during and after the writing of a unit of the monitoring mini-team’s comments as well as of criterion referenced feedback from advisors.
When reviewing your materials you will inevitably find that some criteria questions are not answerable, not valid or not useful and you will decide to delete or modify them.
You might also decide to add criteria that occur to you during the evaluation process.
This ongoing, organic development of evaluation criteria is an important aspect of the evaluation process and is one of the many reasons why you can never just take a checklist from the literature and use it to evaluate any materials.
If you are reviewing materials whose writing you have not participated in we believe it is important to use whatever criteria (if any) which have been developed specifically for those materials by their writers or by the journal or publisher. If no evaluation cri- teria have been developed then we think you should develop criteria yourself which are specific to the materials you are evaluating and then ask the writers or sponsor if they are appropriate. If you are reviewing materials as part of a research project we would recommend developing a set of materials specific criteria and then having your criteria evaluated before use by experts with experience of evaluation.
When developing criteria to drive the writing of a course of materials we believe you need a thorough and detailed set of criteria. This is a time-consuming, demanding but
Materials Development for Language Learning
necessary process. Obviously though you would not need as many criteria if you were developing supplementary materials for your own class or if you were writing a brief review for a journal.
When Trialing Materials
Despite what Amrani (2011) says about publishers no longer relying on trialing for feed- back we hope that the practice of trialing materials before their final publication will continue, as we think it is the most reliable way of gaining information about the effec- tiveness of the materials for their users. Trialing depends on the good will of the teachers invited to trial the materials and therefore asking them to conduct a multi-criteria eval- uation is out of the question. We believe, though, that it is very important that revised versions of a selection of the most important evaluation questions developed during the pre-writing evaluation are used rather than asking the vague, general questions about teacher and learner attitudes, which we have sometimes experienced in trialing. One way of achieving coverage of as many criteria as possible is to give different evaluation questions to different groups of teachers so as not to place too great a demand on their time.
When Selecting Materials
We find that after so many years of evaluating materials we can informally apply our most important criteria to the selection of materials but we would strongly recommend, in any selection procedure for an institution or especially for a nation, that a formal criterion referenced procedure is followed. In our view this means going through the seven-stage procedure recommended for writing and evaluating materials above and comparing the grades and comments for each book evaluated before making a selection.
When Adapting Materials
We believe that formal adaptations for an institution or nation require the formal seven- stage evaluation procedure prior to decisions about adaptation. In such cases, it is important that suggestions for adaptation are included in the comments column of the evaluation table. However, many adaptations are made on the spur of the moment by teachers who realize that a unit of materials they are being asked to use is unlikely to be effective for a particular class. In such a case we would recommend that the following questions are asked and then modifications are made:
To what extent are the materials likely to:
r
Expose the learners to the language in use?r
Be relevant to the lives of the learners?r
Engage the learners affectively?r
Engage the learners cognitively?r
Provide opportunities for the learners to make discoveries about the target language is typically used?r
Provide opportunities for the learners to use the target language for purposeful com- munication?3 Materials Evaluation
Some of our best lessons have resulted from asking these questions of a unit of material and then making last-minute modifications to how we intend to use it (sometimes even on the stairs on the way to the classroom).
For suggestions and examples of informal and impromptu adaptations see Chapter 4,
“Materials Adaptation,” in this volume.
When Using Materials
It can be very revealing to evaluate materials as they are being used either as a teacher using the materials with a class or as an observer watching materials being used. Obvi- ously we cannot observe acquisition taking place and there are many criteria from a pre-use evaluation which would not be answerable in a whilst-use observation. How- ever, in our view we can observe, for example, the extent to which the learners:
r
can follow the instructions without help;r
can understand the “texts” without help;r
can do the activities without help;r
are motivated to experience the “texts”;r
are engaged by the “texts”;r
are motivated to do the activities;r
are engaged by the activities.Each of the above capabilities needs breaking down into specific questions to focus and facilitate a whilst-use evaluation. For example,
r
What approximate percentage of the learners seem to have completed the activity?r
What approximate percentage of the learners seem to have completed the activity successfully?Of course, one major difficulty in observing materials in use is separating evaluation of the materials from evaluation of how the teacher is using them. For example, there might be a number of authentic texts in a unit with great potential for exposure to the target language in use but the teacher only uses one of them and spends most of the lesson doing the drills and artificial exercises from the unit. Or the teacher might add a really interesting readiness activity before asking the students to read a text with the result that the students are affectively engaged whilst reading it. Our advice is that it is best for you or the teacher you are observing to keep to the unit as it is if your main aim is to evaluate the effectiveness of the materials rather than the effectiveness of the teacher’s use of the materials (unless of course the latter is a research focus). We would advise a narrow focus when evaluating materials in use as it is very difficult to notice many different effects at the same time and a narrow focus helps in noticing and recording in greater detail and with more rigor. For example, the focus could be engagement during a task and the observer could use the following questions to “measure” the effectiveness of the materials for the learners:
To what extent did the learners:
r
Stay on task?r
Maintain eye contact with the “texts” they were asked to read or watch as part of the task?r
Maintain eye contact with each other during oral communication phases of the task? Materials Development for Language Learning
r
Maintain proximity to the materials during “text” phases of the task?r
Maintain proximity with each other during oral communication phases of the task?r
Volunteer contributions during pair or group phases of the task?r
Volunteer contributions during plenary phases of the task?After Using Materials
Post-use (or retrospective) evaluation of the effect of materials is the most revealing form of materials evaluation but it is also the least common. This is probably because such an evaluation is inevitably time consuming and demanding of expertise. It is also difficult to make reliable and valid because of the problem of ensuring that the effects measured are as a result of the materials rather than, for example, the teacher’s use of them or the learners’ subsequent experience of the target language in use. In order to gain results of any significance the evaluation would have to be on a large scale, involve many teach- ers with a variety of backgrounds and pedagogic inclinations, involve learners with a variety of objectives and motivations, have access to information about exactly what the teachers and learners did while using the materials and also record the learners’ out- of-class and after-course experience of English. Publishers could undertake such eval- uations but the cost could be high both of conducting the evaluation and of modifying materials already published. Consortia of universities and publishers could undertake such evaluations and it is our hope that they will.
Pre-use and whilst-use evaluations can legitimately and usefully predict and mea- sure the attitudes of the users towards the materials being evaluated as positive atti- tudes would be potential indicators of likely successful effects. However, we believe that post-use evaluations should focus wherever possible on the actual capabilities of the learners who have used the materials either in relation to general ability to use the tar- get language for communication or with reference to the specific performance targets which the materials were being used to help them attain (e.g. getting a good grade on the IELTS examination; being able to study architecture successfully in the medium of the target language; being able to use English safely and effectively as an airline pilot).
Quite often, though, in the few published post-use evaluations we have read, the instru- ments of measurement have not achieved sufficient validity to justify the claims of the evaluators. For example, Hadley (2014, p. 230) claims to have demonstrated empirically that global textbooks “can play an important role in helping…second language learn- ing.” But this claim is based on a study in which a particular global coursebook was used with learners in Japan who were shown to have increased their scores on the coursebook placement test, which consisted of “70 multiple choice items divided into three sections that assess listening…reading…and grammatical knowledge” (p. 222). Can such a placement test provide a valid test of language learning? It certainly cannot be a valid test of communicative competence.
Potentially valid ways of measuring performance effects in our opinion include: