This chapter focuses on the iterative nature of educational design research. Each iteration or cycle of analysis, design and formative evaluation gives the design research team firmer ground and arguments for the intervention the team is working on in order to solve a complex educational problem. The empirical data the team collects during a formative evaluation will not only provide suggestions for improving the intervention, but will also assist in sharpening the accompanying design principles. Proceeding through several of these iterations will end in a final stage of the scientific cycle in which claims of causality can be studied in summative evaluation settings (cf. Nieveen, McKenney & van den Akker, 2006).
In this contribution we concentrated on the research design for each formative evaluation performed within such an iterative or prototyping approach. We elaborated on the research questions, selection of appropriate methods and respondents. We are aware that there is much more to say about formative evaluation in general, and integrated in design research projects in particular. For instance, we could have paid attention to evaluation instruments, data collection, data analysis and reporting. Several helpful books and articles are available to assist in systematically conducting formative evaluation in education (cf. Brinkerhoff, et al., 1983; Flagg, 1990; Tessmer, 1993). Although these sources were not written with the specific needs and wishes of design researchers in mind, they can provide ample inspiration.
References
Brinkerhoff, R.O., Brethouwer, D.M., Hluchyj, T., & Nowakowski, J.R. (1983). Program evaluation: A practitioner’s guide for trainers and eduacators. Boston: Kluwer-Nijhoff.
Creswell, J.W. (2008). Educational research: planning, conducting and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson.
Denscomb, M. (2007). The good research guide for small-scale social research projects.
Maidenhead (UK): Open University Press.
Design-based research collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8.
Flagg, B.N. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum Associates.
Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards. Thousand Oaks, CA: Sage Publications.
Lincoln, Y. & Guba, E.G. (1979). The distinction between merit and worth in evaluation. Paper presented at the Annual Meeting of the Evaluation Network (5th, Cincinnati, OH,
September 24-26, 1979) retrieved October 20, 2008, http://www.eric.ed.gov/ERICDocs/
data/ericdocs2sql/content_storage_01/0000019b/80/32/b5/c8.pdf
Linn, M., Davis, E.A., & Bell, P. (2004). Internet environments for science education. Hillsdale:
Lawrence Erlbaum.
McKenney, S. Nieveen, N. & Akker, J. van den (2006). Design research from a curriculum perspective. In J. van den Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 67-90). London: Routledge.
Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis: An expanded source book.
London: Sage.
Mills, G., Gay, L.R., Airasian,P. & Airasian, P.W. (2008). Educational research: competencies for analysis and applications. Upper Saddle River, NJ: Prentice Hall.
Monk, A., Wright, P., Haber, J., & Davenport, L. (1993). Improving your human- computerinterface: A practical technique. New York: Prentice Hall.
Moonen, J. (1996). Prototyping as a design method. In Tj. Plomp, & D.P. Ely (Eds.),
International encyclopedia of educational technology (pp. 186-190). Oxford: Pergamon.
Nieveen, N.M. (1997). Computer support for curriculum developers: A study on the potential of computer support in the domain of formative curriculum development, Doctoral dissertation. Enschede: University of Twente.
Nieveen, N. (1999). Prototyping to reach product quality. In J. van den Akker, R. Branch, K.
Gustafson, N. Nieveen, and T. Plomp (Eds.), Design approaches and tools in education and training (pp. 125–36). Dordrecht: Kluwer Academic Publishers.
Nieveen, N.M., & Akker, J.J.H. van den (1999). Exploring the potential of a computer tool for instructional developers. Educational technology research & development, 47(3), 77-98.
Nieveen, N., McKenney, S., & Akker, J. van den (2006). Educational design research: The value of variety. In J. van den Akker, K. Gravemeijer, S. McKenney, and N. Nieveen (Eds.), Educational design research (pp. 151-158). London: Routledge.
Reeves, T. (2000). Enhancing the worth of instructional technology research through
“design experiments” and other developmental strategies. Retrieved Oct. 20, 2006 from http://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf
Rossi, P.H., Freeman, H.E. & Lipsey, M.W. (1999). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.
Scriven, M. (1967). The methodology of evaluation. In R.W. Tyler, R.M. Gagné, & M. Scriven (Eds.), Perspectives of curriculum evaluation. AERA Monograph series on curriculum evaluation. nr.1. Chicago, MI: Rand McNally.
Scriven, M. (1991). Beyond formative and summative evaluation. In M.W. McLaughlin, & D.C.
Phillips (Eds.), Evaluation and education: At quarter century (pp. 19-64). Chicago:
University of Chicago.
Shneiderman, B. (1992). Designing the user interface: Strategies for effective human- computer interaction. Reading, MA: Addison-Wesley.
Smith, M.F. (1991). Software prototyping: Adoption, practice and management. London:
McGraw-Hill.
Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. London: Kogan Page.
Van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. Branch, K. Gustafson, N. Nieveen, and T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1–15). Dordrecht: Kluwer Academic Publishers.
Van den Akker, J. (2003). Curriculum perspectives: An introduction. In J. van den Akker, W. Kuiper & U. Hameyer (Eds.), Curriculum landscapes and trends (pp. 1-10). Dordrecht:
Kluwer Academic Publishers.
Wademan, M. (2005). Utilizing development research to guide people capability maturity model adoption considerations. Unpublished doctoral dissertation.
Syracuse: Syracuse University.
Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational training research & technology, 43(3), 29-48.
6. References and Sources on Educational Design Research
Tjeerd Plomp and Nienke Nieveen
Introduction
This bibliography has been compiled to support researchers and graduate students in getting access to key publications on design research. We do not claim that the selection of sources included in this chapter is complete and exhaustive – it is coloured by our
background and bias as well as our knowledge and familiarity with publications. Important criteria for us to include titles in this bibliography are (i) proven usefulness of sources for our own work, and (ii) representing important perspectives and groups that are (or have been) actively working in this domain.
In the first section we present just an overview of relevant sources available. This is followed by a section in which we present the structure and content of the excellent website ‘Design-based Research EPSS’ (http://projects.coe.uga.edu/dbr/index.htm)– created by Instructional Technology Ph.D. students at The University of Georgia (last update November 2006). Given the quality and completeness of this website (at least till November 2006), we decided to introduce it in a separate section and in the other sections of this chapter we will refer to parts of this website, but also introduce a number of other sources.
In the following two sections we point the reader to selected journal articles and book chapters on the concept and methodology of design research and on design research in domains such as curriculum, instructional technology, and the learning of reading and writing, mathematics and science. In the final section, we list the URLs of a number of doctoral theses that have been defended in The Netherlands utilizing design research as a research approach.
As stated, our selection is coloured by our bias and experience, but all these publications refer to a wide range of writings on design research and we trust that they therefore serve as a useful introduction to the reader.