Since most design languages and systems are recent as of 2009, evaluation criteria and methods may yet be open to debate. Most evaluation schemes are designer-oriented, i.e. proposed by people who invent design languages and implement design systems.
Botturi (2005:335) proposed an issues- and elements-based evaluation framework for instructional design languages:
“
1. Issues are critical aspects that should be considered in the definition of the experimental setting. They are: context sensitivity, eclectic benefits, course quality, and time. 2. Elements are indications for the identification of key variables in the study. They are: impact on sub-activities, communication events, institutional changes, and expressive power.”
LeJeune et al. (2009) summarize the following vital issues and challenges:
• Comprehensibility: how can EMLs be made usable for educational practitioners ? (Pernin & Lejeune, 2006;
Hernãndez-Leo et al., 2007) ?
•• Pedagogical neutrality: how can an EML realize one unified, pedagogical neutral notation for supporting a large variety of pedagogically sound scenarios (Miao et al, 2005, Miao et al., 2008) ?
• Flexibility: how can EMLs support design of wellsupported, but flexible environments (Dillenbourg, 2002;
Dillenbourg & Tchounikine, 2007) ?
•• Interoperability: how can EMLs build on existing learning platforms and contents (Ardito et al., 2006).
To that we would like to add another most important one: To what extent is the system available, operationable, documented and maintained ?
Finally, we would like to argue that such top-down evaluation schemes should be complemented by idiographic methods, e.g. repertory grid technique based analysis.
Links
• Digital repository of Research, Technology Development and Valorisation outcomes of the Open Universiteit Nederland [5]
• Revue international des technologies en pédagogie universitaire [6] volume 4 - 2007 - numéro 2.
• Pernin J-P., Godinet H. (2006), Actes du colloque Scénariser l'enseignement et l'apprentissage : une nouvelle compétence pour le praticien ? [7]. INRP, Lyon, avril 2006.
Bibliography
•• Ardito, C.; M.F. Costabile, M. De Marsico, R. Lanzilotti, S. Levialdi, R. Roselli, and T. Rossano, (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, Springer, vol. 4, no. 3, pp. 270-283.
•• Brabazon, T (2002) Digital Hemlock: Internet Education and the Poisoning of Teaching, Sydney: University of New South Wales Press
• Abdallah, F., Toffolon, C., and Warin, B. 2008. Models Transformation to Implement a Project-Based Collaborative Learning (PBCL) Scenario: Moodle Case Study. In Proceedings of the 2008 Eighth IEEE international Conference on Advanced Learning Technologies - Volume 00 (July 01 - 05, 2008). ICALT. IEEE Computer Society, Washington, DC, 639-643. DOI= http://dx.doi.org/10.1109/ICALT.2008.174
• Boot Eddy W.; Jon Nelson, Jeroen J.G. van Merriënboer, Andrew S. Gibbons (2007). Stratification, elaboration and formalisation of design documents: Effects on the production of instructional materials, British Journal of Educational Technology 38 (5), 917–933. doi:10.1111/j.1467-8535.2006.00679.x [8]
• Botturi, L. (2005). A Framework for the Evaluation of Visual Languages for Instructional Design: the Case of E2ML. Journal of Interactive Learning Research. 16 (4), pp. 329-351. Norfolk, VA: AACE. Abstract/PDF [9]
• Botturi, L. (2006). E2ML. A visual language for the design of instruction. Educational Technologies Research &
Development, 54(3), 265-293. Abstract/PDF [10] (Access restricted)
• Botturi, L. and S. Todd Stubbs (eds.) (2007). Handbook of Visual Languages for Instructional Design: Theories and Practices, Information Science Reference. ISBN 1599047292, Google books preview [11]
• Botturi, L., Derntl, M., Boot, E., & Figl, K. (2006). A Classification Framework for Educational Modeling Languages in Instructional Design. Proceedings of The 6th IEEE International Conference on Advanced Learning Technologies, 1216-1220 PDF [12] (also here [13]).
• Botturi, L., Stubbs, T. (eds.) (2007). Handbook of Visual Langauges in Instructional Design: Theories and Pratices. Hershey, PA: Idea Group. PDF Flyer [14] - Some chapters are free on Scribd for screen reading [15]. The best reader currently, but fairly expensive - Daniel K. Schneider 14:48, 28 August 2008 (UTC). ISBN
978-1-59904-729-4
•• Brady, Aoife; Owen Conlan, Vincent Wade, and Declan Dagger (2008). Personalised Learning Objects, in Nedjl, Wolfgang; Judy Kay, Pearl Pu, Eelco Herder (eds.) Adaptive Hypermedia and Adaptive Web-Based Systems, Proceedings of the 5th International Conference on Adaptive Hypermedia . Springer. ISBN 3540709843
• Breuker, J., Muntjewerff, A., and Bredewej, B. (1999) "Ontological modeling for designing educational systems"
I* PALO [16] n Proceedings of the AIED 99 Workshop on Ontologies for Educational Systems, Le Mans, France.
IOS Pressp
• Burgos, D., Arnaud, M., Neuhauser, P., Koper, R. IMS Learning Design : la flexibilité pédagogique au service des besoins de la e-formation. In La Revue de l'EPI. France: L'association Enseignement Public et Informatique [17]. Available at and http://hdl.handle.net/1820/470 [18].
• Conole, Gráinne and Karen Fill (2005). A learning design toolkit to create pedagogically effective learning activities. Journal of Interactive Media in Education (Advances in Learning Design. Special Issue, eds. Colin Tattersall, Rob Koper), 2005/08. ISSN:1365-893X Abstract [17] (PDF/HTML open access)
•• Del Cid, Jose Pablo Escobedo, Luis de la Fuente Valentín, Sergio Gutiérrez, Abelardo Pardo, Carlos Delgado Kloos (2007). Implementation of a Learning Design Run-Time Environment for the .LRN Learning Management System. Journal of Interactive Media in Education (Adaptation and IMS Learning Design. Special Issue, ed.
Daniel Burgos), 2007/07. ISSN:1365-893X
• Dessus, Philippe et Schneider, Daniel Scénarisation de l'enseignement et contraintes de la situation, In J.-P. Pernin
& H. Godinet (2006). (Eds.), Colloque Scénariser l'enseignement et l'apprentissage : une nouvelle compétence
pour le praticien ? (pp. 13-18). Lyon : INRP. PDF [19]
• Dillenbourg, P. & P. Tchounikine (2007). Flexibility in macro-scripts for computer-supported collaborative learning, Journal of Computer Assisted Learning, 23 (1). DOI 10.1111/j.1365-2729.2007.00191.x [20]
• Dillenbourg, Pierre & Patrick Jermann (2007), Designing Integrative Scripts. In: Fischer, F., Kollar, I., Mandl, H., Haake, J. (eds.): Scripting Computer-Supported Collaborative Learning. Cognitive, Computational, and
Educational Perspectives. Springer, New York.
•• Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. In P. A. Kirschner (Ed). Three worlds of CSCL. Can we support CSCL, Heerlen, Open Universiteit Nederland, 2002, pp. 61-91.
•• Dillenbourg P. and P. Tchounikine (2007). Flexibility in macro-scripts for CSCL. Journal of Computer Assisted Learning, vol. 23, no. 1, 2007, pp. 1-13.
•• Fowler, M., UML distilled: A brief guide to the standard object modeling language. Addison Wesley, Boston, MA, 2003.
• Friesen, Norm, (2004). A Gentle Introduction to Technical E-learning Standards, Canadian Journal of Learning and Technology Volume 30(3) Fall / automne 2004. HTML [21] , retrieved 17:52, 5 October 2006 (MEST).
• Friesen, Norm, (2004). The E-learning Standardization Landscape, HTML [22] , retrieved 17:52, 5 October 2006 (MEST).
• Gibbons, A. S. (2003). What and how designers design? A theory of design structure. TechTrends, 47(5), 22–27.
PDF [23] (Access restricted)
• Gibbons, A. S., Nelson, J. & Richards, R. (2000). "The nature and origin of instructional objects" In D. A. Wiley (Ed.)," The Instructional Use of Learning Objects". Bloomington: Association for Educational Communications and Technology.
• Gibbons, Andrew, S. and Erin K. Brewer, (2005) “Elementary principles of design languages and design notation systems for instructional design”. In J.M. Spector, C. Ohrazda, A. Van Schaack, and D. Wiley (Eds.), Innovations to instructional technology: Essays in honor of M. David Merrill, Lawrence Erlbaum Associates, Mahwah NJ, pp.
111-129.
• Giesbers, B., van Bruggen, J., Hermans, H., Joosten-ten Brinke, D., Burgers, J., Koper, R., & Latour, I. (2007).
Towards a methodology for educational modelling: a case in educational assessment. Educational Technology &
Society, 10 (1), 237-247. PDF [24]
• Griffiths, D. et Blat, J. (2005). The role of teachers in editing and authoring units of learning using IMS learning design. Advanced Technology for Learning, 2(4). PDF [25] and PDF [26]
•• D. Hernãndez-Leo, A. Harrer, A., J.M. Dodero, J. I. Asensio-Pérez, and D. Burgos (2007). A Framework for the Conceptualization of Approaches to "Create-by-Reuse" of Learning Design Solutions, Journal of Universal Computer Science, vol. 13, no. 7, 2007, pp. 991-1001.
•• Kobbe, L. Weinberger, A. Dillenbourg, P. Harrer, A. Hämäläinen, R. Hàkkinen, P. Fischer, F. (2007). Specifying computer-supported collaboration scripts, International Journal Of Computer-Supported Collaborative Learning, 2 (2-3) 211-224.
• Koper R. (2001) "Modelling Units of Study from a pedagogical perspective: The pedagogical metamodel behind EML" Technical Report OUNL June, 2001 http://eml.ou.nl
• Koper, R. (2000) "From change to renewal: Educational technology foundations of electronic learning environments" Technical Report, Open University of the Nederland (OUNL) http://eml.ou.nl
• Koper, R. (2002) Educational Modelling Language: adding instructional design to existing specifications, Workshop Standardisierung im eLearning 10./11.4.2002 an der Universität Frankfurt/Main, PDF [1]
• Koper, R. and Manderveld, Jocelyn (2004). Educational modelling language: modelling reusable, interoperable, rich and personalised units of learnings, British Journal of Educational Technology, Vol 35 No 5 2004, 537-551.
• Koper, R., Rodr¡guez-Artacho, M., Rawlings, A., Lefrere, P., van Rosmalen, P. (2002) "Survey of Educational Modeling Languages" Technical Report of the CEN/ISSS Learning Technologies Workshop Available On-Line:
HTML [27] (dead link, it's amazing that the european standardization body can't keep URLs online ...)
• Laforcade, P., Zendagui, B., and Barré, V. 2008. A Domain-Specific-Modeling Approach to Support
Scenarios-Based Instructional Design. In Proceedings of the 3rd European Conference on Technology Enhanced Learning: Times of Convergence: Technologies Across Learning Contexts (Maastricht, The Netherlands, September 16 - 19, 2008). P. Dillenbourg and M. Specht, Eds. Lecture Notes In Computer Science, vol. 5192.
Springer-Verlag, Berlin, Heidelberg, 185-196. DOI= http://dx.doi.org/10.1007/978-3-540-87605-2_21
• LittleJohn, Allison (2005), From learning objects to learning design, AsciLite Newsletter. HTML [28]
• Maglajlic S., Maurer H., and Scherbackov N. (1998) "Separating structure and content, authoring Educational web applications" In Proceedings of the ED-MEDIA & ED-TELECOM 98., pages 880-884, 1998.
• Martel C., Vignollet L., Ferraris C., David J.P., Lejeune A. (2006), Modeling collaborative learning activities on e-learning platforms, ICALT 06, (PDF [29])
• Martel Christian, Laurence Vignollet, Christine Ferraris, Guillaume Durand (2006), LDL: a Language to Model Collaborative Learning Activities, ED-MEDIA 2006 PDF Preprint [30]
•• Merrill, M. D. (2001) "The instructional use of learning objects, chapter "Knowledge objects and mental-models"
D. Wiley, Ed. AIT Publishers ISBN: 0-7842-0892-1
•• Miao, Y.; K. Hoeksema, H.U. Hoppe, and A. Harrer (2005). CSCL Scripts: Modelling Features and Potential Use.
Computer Supported Collaborative Learning (CSCL), Taipei, Taiwan, 2005, pp. 423-432.
• Miao, Y; M. van der Klink, J. Boon, P. Sloep, and R. Koper (2008). Enabling Teachers to Develop Pedagogically Sound and Technically Executable Learning Designs, Ten-Competence publications, http://hdl.handle.net/
1820/1605.
• Nodenot Thierry & Laforcade Pierre CPM: a UML Profile to design Cooperative PBL situations at didactical level, Proceedings of the Sixth International Conference on Advanced Learning Technologies (ICALT'06) 0-7695-2632-2/06 PDF [31]
• Pantano Rokou, F., Rokou, E., & Rokos, Y. (2004). Modeling Web-based Educational Systems: process Design Teaching Model. Educational Technology & Society, 7 (1), 42-50. PDF [32]
• Paquette, G. (2006). Introduction à la spécification IML-LD d’une perspective d’ingénierie pédagogique.
Récupéré le 7 septembre 2007 du site IDLD (Déploiement et implantation de la spécification pour les designs d’apprentissage), http://www.idld.org/Methodology/tabid/174/Default.aspx
• Parrish, Patrick, E. (2007). Aesthetic principles for instructional design, Educational Technology Research and Development (ETRD), http://dx.doi.org/10.1007/s11423-007-9060-7.(Abstract/HTML/PDF)
• Pernin, J.-P. et Godinet, H. (2006). Scénariser l’enseignement et l’apprentissage : une nouvelle compétence pour le praticien? Actes du colloque Scénario 2006. INRP [33]
•• Pernin,J-P. and A. Lejeune, (2006) Learning design: models for computers, for engineers or for teachers, Workshop Learning Networks for Lifelong Competence Development, Project TEN-Competence, Sofia, Bulgaria, 2006.
• Rawlings, Adrian; Peter van Rosmalen, Rob Koper,Miguel Rodríguez-Artacho, and Paul Lefrere (2002). Report on Educational Modeling Languages (October 2002) [34] . Probably the best overview
• Reigeluth, C. M. & Nelson, L. M. (1997). A new paradigm of ISD? In R. C. Branch & B. B. Minor (Eds.), Educational media and technology yearbook (Vol. 22, pp. 24-35). Englewood, CO: Libraries Unlimited.
•• Ritter, S. and Suthers, D. (1997). "Technical Standards for Education" Working Paper, Educational Object Economy site, The EOE Foundation."
•• Robson, R. (2000). "Report on Learning Technology Standards", in J. Bourdeau and R. Heller, Eds., Proceedings of ED-MEDIA'00, the Association for the Advancement of Computing Education, Charlottesville, Virginia.
• Rodriguez-Artacho, M. (2002) "PALO Language Overview" Technical Report STEED Project (LSI Dept.
UNED) February, 2002. HTML [35]
• Rodr¡guez-Artacho, M. and M.F. Verdejo (2001) "Creating Constructivist Learning Scenarios Using an Educative Modelling Language" in Proceedings of the IEEE Frontiers in Education 2001 Conference, Reno NV Oct 2001.
Available on-line at PDF [36]
• Rodríguez-Artacho, M., & Verdejo Maíllo, M. F. (2004). Modeling Educational Content: The Cognitive Approach of the PALO Language. In Journal of Educational Technology & Society, 7 (3), 124-137. PDF [37]
•• Sampson, D., Karampiperis, P. et Zervas, P. (2005). ASK-LDT: A Web-based learning scenarios authoring environment based on IMS learning design. Advanced Technology for Learning, 2(4), 207-215.
• Savard, E. (2007). Scénario 2007, le 2e colloque international sur les scénarios pédagogiques. html [38] (Profetic)
• Péricles Sobreira, Pierre Tchounikine, Table-based representations can be used to offer easy-to-use, flexible, and adaptable learning scenario editors, Computers & Education, Volume 80, January 2015, Pages 15-27, ISSN 0360-1315, http://dx.doi.org/10.1016/j.compedu.2014.08.002.
• Streng, S., Stegmann, K., Fischer, F., & Hussmann, H. From Graphical Learning Designs To
Computer-Supported Collaboration Scripts: A Rapid Development Process, ICTEL 2011. PDF preprint [39]
• Süss, C., Freitag, B and P. Broessler (1999) "LMML: Metamodelling for Web-based Teachware Management" in Proc. Intl. ER '99 Paris, France LNCS 1727 Springer Verlag http://daisy.fmi.uni-passau.de/db/literatur.
php3?key=SFB99 (dead link)
•• Teege, Gunnar; Jürgen Koch, Pamela Tröndle, Wolfgang Wörndl, Johann Schlichter (2000). ModuVille:
Komponenten für virtuelle WWW-basierte Lehrveranstaltungen, PIK - Praxis der Informationsverarbeitung und Kommunikation, pp. 148-155. (this is a TargeTeam/TeachML publication).
•• Villiot-Leclercq, E., David, J., and Lejeune, A. 2006. Expressing Learning Scenarios with Computer Independent Models. In Proceedings of the Sixth IEEE international Conference on Advanced Learning Technologies (July 05 - 07, 2006). ICALT. IEEE Computer Society, Washington, DC, 520-522.
• Waters, Sandie, H. & Andrew, S. Gibbons (2004). Design languages, notation systems, and instructional technology: A case study: Educational Technology Research and Development, 52(2), 57-69. PDF [40] (Access restricted)
• Wecker, C.; K. Stegmann, F. Bernstein, M. J. Huber, G. Kalus, S. Rathmeyer, I. Kollar and F. Fischer, “S- COL: A Copernican turn for the development of flexibly reusable collaboration scripts”, International Journal of Computer-Supported Collaborative Learning, 5(3), 2010, pp. 321-343.
• Wilson, S. (2001) "Europe Focuses on EML's" Report from CETIS Research Centre, UK. HTML [41]
Acknowledgement: This article or part of this article has been written during a collaboration with the EducTice [7] group of INRP [8], which attributed a visiting grant to DKS in january 2009.