Effects of two modes of pedagogical agent on learning among students of Arabic at Centre for Languages,
International Islamic University Malaysia
Dr. Mohd Feham Md. Ghalib Quranic Language Division
Centre for Languages and Pre-university Academic Development International Islamic University Malaysia
Tel: 03-61964906 Fax: 03-61964865 Email: [email protected]
Abstract
This study examined the effects of a pedagogical agent (PA) with different levels of personalization in a web courseware on the learning of thematic materials.
Two modes of an interactive multimedia web courseware on the theme of the holy city of Al-Quds were systematically designed and developed. While the first mode used a personalized version of the pedagogical agent (PPA), the second mode used a non-personalized PA (NPPA). The personalization elements include an integration of social cues, dialogue conversations, and indirect feedback. The rationale of adopting pedagogical agents in a web learning environment is based on Vygotsky’s social learning theory (1978) , communication theory (Reeves & Nass, 1996), cognitive multimedia learning theory (Mayer, 2001), and cognitive flexibility theory (Spiro et al., 1992). The study found that PPA mode outperformed NPPA mode in an open-ended question in the administered test. The study suggests that PPA mode was more effective in increasing learning engagement and interest for the overall understanding and comprehension of the Islamic culture and civilization information integrated in the web courseware.
Introduction
However, several studies have alerted that the constructivist nature of the Internet content may lead to difficulties and become learning impediments if it is left to the students without proper instructional guidance (Allen, 2003; Toh, 2004; Tuovinen, 2001). This guidance acts as a facilitator to ensure that the student will be able to efficiently choose a suitable strategy for any learning tasks or activities at any time without compromising the goals of the required learning.
In this respect, the role of an emerging tool, namely the pedagogical agent (PA) offers a feasible and enticing approach to tackle this issue. The significant functionality of the PA has been validated in several studies (Mayer, Dow & Mayer, 2003; Moreno & Mayer, 2004; Mayer, Sobko
& Moutone, 2003). Though this tool has been around for some time, the usability potential of the tool has yet to be discovered and still in its infancy. The current study argues that adding more personalized elements onto the PA will accelerate the process of internalizing the learning materials due to the factors relatively referred to learning engagement and preparedness to complete the tasks or assigned activities.
Statement of Problem
Recently personalized instructions in a computer based learning environment have been of interest among educational technologists (Martinez, 2001; Mayer et al., 2004; Mayer, Sobko &
Mautone, 2003; Moreno & Mayer, 2000). In fact, learners are always treating computers as their personalized social companions (Reeves & Nass, 1996).
In this respect, the need to have more personalized instructions for accelerating the process of acquiring knowledge is important. The integrity of learning materials designed in accordance to the principles of multimedia personalization is crucial to ensure that students are able to engage themselves with the learning process (Allen, 2003). Additionally, a personalized instructional design will be able to guide the students to choose a correct learning strategy in line with their
learning tasks and activities. It has been found that personalization in instructions has been studied sporadically by researchers and educationists with each research focusing on a small or minor component of this aspect. As such, this study intends to go a step further in pioneering the combination of these personalization elements and examine whether the results would be supportive and in line with the previous studies conducted in this area.
The current study argues that there are three personalization elements: 1) multimedia principle of personalization 2) integration of social cues, and 3) indirect corrective feedback, serve as an accelerating factor for learning engagement and learning management strategies and thus affect the outcome of the learning process i.e. learning attainment and performance.
Theoretical Framework
The theories and models underlying this study are divided into two sections. Firstly, the theories and models that support the use of the web and the agent character from a pedagogical rationale and secondly, the practical model related to the instructional representation of this study. The theories supporting web usage and social PAs employment are social learning theory (Vygotsky, 1978), communication theory (Reeves & Nass, 1996) and cognitive multimedia learning theory (Mayer, 2001). The nature of teaching-learning through the web materials is explained by the application of the cognitive flexibility theory (Spiro et al., 1992).
Research Question and Hypothesis
From the presented theoretical rationale the research question was formulated to investigate whether students taught via a personalized pedagogical agent perform significantly higher than students taught via a non-personalized pedagogical agent in terms of learning gains and performance. This resulted a hypothesis which suggests that students taught via a
personalized pedagogical agent will perform significantly better than students -taught via a non- personalized pedagogical agent.
Research Procedures
This study was a quasi-experimental research that was designed to investigate the effects of interaction with either a PPA or a NPPA on language performance. The independent variable of the study is the two modes of presentation: 1)web courseware with a personalized pedagogical agent version (PPA) and 2) web courseware with a non-personalized pedagogical agent version (NPPA). The dependent variable is the Language test performance. There were 96 students participated in the study throughout the period of experiment.
The content of the instructional materials for this study was oriented to Islamic culture. The topic of the city of Al-Aqsa which is located in Palestine was selected to be the main theme of these materials. The elements included in this theme were the art and architectural structures of this historical city, such as mosques, minarets, and sources of water termed as sabel in Arabic. Some information related to the historical aspects of this city were also presented to enrich the information given about this historical city. The topic was chosen due to its Islamic cultural perception which is propagated and upheld by the university environment and represents as one of the topics integrated in the Arabic syllabus currently taught at the International Islamic University, Malaysia.
Instrument
In order to collect the data for the study a language performance test was used to assess the students’ language performance after using the web-based courseware.
Data collection
Four instructors were chosen to assist the researcher on collecting the data. Prior to the procedures, they were trained and given hands-on experience in working with the web courseware. They were also given a lesson plan containing a list of detailed activities that should be conducted for each of the class sessions.
Accordingly, the answer scripts of the performance test were corrected and marked. For marking the open-ended question component of the test, two instructors with over 10 years of teaching experience were chosen. This was to reduce the element of bias in the assessment process. The clear marking guidelines were given to both instructors with the content allocated the highest score of 6 points and 2 points for each of the language styles, and the spelling and grammar respectively.
Data analyses
The data collected were analyzed using a computer program Statistical Package for The Social Sciences (SPSS) for Windows version 11.5. Thus, the following procedures were taken in analyzing the data for the purpose of either substantiating or refuting the formulated hypotheses.
1. The data were entered and coded accordingly. The frequency tabulation was used to map the subjects based on the modes of presentation. Then, based on the central tendency principle, the mean scores of the learning gains (Post-test - Pre-test), the standard deviation values, and the variance equality were measured.
2. A One-way ANOVA was used to determine if the differences found in the mean scores were significant between the two modes of presentation.
Details on the groups for both modes, PPA and NPPA are tabulated in Table 1.
Table 1 Distribution of Groups Based on The Modes of Presentation
Modes of Presentation Frequency Percent
PPA 40 41.7
NPPA 56 58.3
Total 96 100.0
The mean scores of both the two variables are tabulated in Table 2. Additionally, details on the results for components incorporated in the learning gains are also given. For the learning gains, the components include agree-disagree, matching, recall, and open-ended. Table 2 shows that the mean score of the learning gains for PPA (learning gains
µ µ µ µ
= 10.38) are higher than the mean score for NPPA (learning gainsµ µ µ µ
= 8.06). A similar result can also be observed from the rest of the entailed components of the learning gains including, agree-disagreeµ µ µ µ
= 4.38, matchingµ µ µ µ
= 1.05, recallµ µ µ µ
= 3.43, and open-ended questionµ µ µ µ
= 1.76.Table 2 The Mean Scores and Standard Deviations For Two Modes of Presentation
Modes
Of Presentation Mean
Std.
Deviation N
Learning Gains 10.38 7.25
• Agree-disagree 4.38 5.10
• Matching 1.05 1.85
• Recall 4.35 4.14
Personalized (PPA)
• Open-ended 1.76 1.57
40
Learning Gains 8.06 7.60
• Agree-disagree 4.02 5.66
• Matching .34 1.82
• Recall 3.43 3.66
Non-personalized (NPPA)
• Open-ended .80 1.63
56
The test of statistical significance for differences observed in the means was determined using the results of independent t-test statistical procedure. The result yielded the values of F1,90 = 1.810, Mean Square = 1.645, and p = 0.182 have been reported for the two modes of presentation. This indicates that the differences in the means between both modes were not statistically significant.
The analysis explored the components of the language performance test by comparing the mean scores of the components listed under the learning gains for both two modes of presentation exhibited the higher achievement of the PPA group (agree-disagree
µ µ µ µ
= 4.38, matchingµ µ µ µ
= 1.05, recallµ µ µ µ
= 4.35, and open-ended questionµ µ µ µ
= 1.76) and NPPA group (agree-disagreeµ µ µ µ
= 4.02, matchingµ µ µ µ
= 0.34, recallµ µ µ µ
= 3.43, and open-ended questionµ µ µ µ
= 0.80) as tabulated in Table 2.Table 3 shows the detailed One-way ANOVA results on the components under the learning gains in a summarized format. Based on the comparison between the two modes, the mean score for the open-ended question component yielded the significant outcome from the test with F1,90 = 8.372, Mean Square = 1.029, p = 0.005, and Eta2 = 0.085. This means that the modes of presentation statistically explained 8.5 % of the open-ended question variance. The other three components, however, did not show any statistical significance in the mean score differences.
Table 5.11 Summary of One-way ANOVA of The Mean Scores of The Components (Agree-disagree - AG, Matching, Recall, and Open- ended Question - OEQ) Under The Learning Gains (Post-test - Pre- test) For The Modes of Presentation
Type III Sum of Squares
Mean
Square F Sig.
Partial Eta Squared
Observed Power(*)
AG 6.130 6.130 .229 .633 .003 .076
Match 12.984 12.984 3.769 .055 .040 .484
Recall .006 .006 .015 .903 .000 .052
OEQ
Source df for MODE = (1, 90),
1.029 1.029 8.372 .005 .085 .817
* Computed using alpha = .05
Thus, the results of these analyses were summarized as follows:
• There is no significant difference between the mean scores of the learning gains for both students taught via a PPA and via a NPPA.
• Students taught via a PPA performed significantly better than students taught via a NPPA in the open-ended question component of the learning gains.
Though the agree-disagree, matching, and recall components demonstrated no significant outcomes, the results for the open-ended question component were statistically significant.
Thus, in overall, this hypothesis is partially supported.
Discussion
The study reported a higher mean score for the learning gains and program rating score for the PPA mode when compared to the NPPA mode. These mean score differences, however, are not statistically significant. Therefore, the study is not able to provide with clear evidence on the advantages of applying personalized pedagogical over non-personalized PAs. Though this may be the case, detailed analysis on the components of the learning gains uncovered useful insights on the significance of PPA use. From four incorporated components (agree-disagree, matching, recall, and open-ended question) of the learning gains, the open-ended question component produced a significant outcome which indicates that the PPA group significantly outperformed the NPPA group in this aspect.
The significant advantage of applying personalized PAs has been substantially supported by several research conducted in the field of educational psychology (Atkinson, 2002; Craig, Gholson & Driscoll, 2002; Mayer, Dow & Mayer, 2003; Mayer, Sobko & Moutone, 2003; Moreno
& Mayer, 2004;). Thus, the results of present study are consistent with the results of their research.
For some individuals, online learning through the Internet could be a daunting experience.
Though, the Internet has been popular among youngsters and adults alike, its role as a platform for distributing knowledge and learning-teaching materials is still considered relatively new. The popular application and usage of the Internet facility tend to be focusing only on communication and leisure-type activities, such as chatting, emailing, blogging, and online gaming. Therefore, a thorough and planned training experience in using the Internet as a source of disseminating knowledge is necessary prior to venturing into the online learning bandwagon.
Given the fact that learning via online needs continuous training and support, the findings of the study would be different if a longer period of time was awarded to the learners in interacting with the materials. The study, however, only allowed a period of ten hours for the subjects to interact with the learning materials within three consecutive weeks. Therefore, a longer duration of the treatment might produce different outcomes.
It may also be inferred that the subjects were influenced by the attractive elements and interactive features of PAs. This could be seen form the high scores for the program rating sheet for both groups. The standard deviation values indicated that the differences existing among the learners are hardly apparent and even, unnoticeable. From this observation, the persona effect of the animated character is apparently observed, particularly, in an environment where there is a strong tendency of users to treat electronic computer gadgets as their social companions (Reeves & Nass, 1996; Laurel, 1991).
The fact that only the open-ended question component showed significant outcome needs to be investigated further. Though, this finding is consistent to the previous studies on PAs, the significance suggests that the PPA mode assists learners in internalizing the holistic picture of learning materials. As for the extended information and supportive points, a longer period of time is required for these details to be remembered and thus, acquired by the learner.
The support of the open-ended question component which is in favor of the personalized version of PA indicates impacts of persona and interest on learning (Schraw, 1998; Harp &
Mayer, 1997) and contradicts to coherence principle of multimedia learning (Mayer, 2001). It also supports the assumption of cognitive flexibility theory (Spiro et al., 1992) against cognitive load theory (Sweller & Chandler,1991; 1994) where the learning should be conducted in a natural setting without oversimplification. The interrelated components and details should be presented in their natural form in order to maintain the integrity of the acquired knowledge.
The results of the study suggest a need to investigate further the effects of the personalized PAs on learning via online. This study only provides support of the personalized over the non- personalized PA. Future research should emphasize on higher levels of personalization and conduct the study for a longer period.
The present study is confined to specific elements related to task, theme, and learners’
characteristics. Future research should investigate future effects of personalized PA on other types of learning task, for instance, those which challenge users to complete questions in a problem-solving environment. The theme used in the study also could be extended to others, such as scientific and technical based lessons such as biology, physics, architecture, and information technology. In relation to the learners’ characteristics, the research could be done in other types and levels of knowledge experiences. This study only investigated learners at the advanced level of language proficiency.
The rapid development of enhancements in the field of information technology and communication presents the need to extend the experience of interacting with PAs from various multimedia enabled gadgets, such as personal digital assistant (PDA), mobile hand phones, and smart hand phones (a combination of both PDA and mobile hand phone). With the availability of larger data transfer bandwidth and more affordable communication equipment, research in this field should investigate their influence and impact on the teaching and learning process.
Finally, some evidence from the study of the speaker’s language abilities should encourage researchers and linguists alike to investigate further other linguistic elements such as dialects, rhetoric, syntax, and morphology. Dialogues that promote creativity, critical thinking, and argumentation are of equal importance and may be investigated. The present study, however, only included a difference of dialogue versus monologue conversation. Linguistic features and cognitive language styles were not covered in the study.
Conclusion
Carroll (1991) notes that some of remarkable features of a computer user interface that have been developed and embraced by end users were incorporated without any assistance from research in the field of educational psychology. One of these remarkable features is a PA.
Therefore, the researcher hoped than this study sheds some lights in the understanding of PA properties and functionality, particularly, in the web teaching-learning environment.
In short, the study suggests that PPA mode which includes three elements of personalization (social cues, dialogue conversations, and indirect feedback) was moderately effective in increasing learning engagement and interest for the understanding and comprehension of the web courseware in a holistic picture. This can be concluded from the finding of the open-ended question component analysis where the mean score of the PPA group is significantly higher than the NPPA group. Therefore, eye movements, gestures, motions, facial expressions, dialogue conversation, indirect feedback are of important elements that affect positively on the learning outcome especially for a longer period.
List of Bibliography
Allen, M. (2003). Guide to E-learning: Building Interactive, Fun, and Effective Learning Programs for any Company, New Jersey: John Wiley & Sons.
Atkinson, R.K. (2002). Optimizing learning from examples using animated pedagogical agents, Journal of Educstional Psychology, 94, 2, pp. 416-427.
Carroll, J. M. (Ed.) (1991). Designing Interaction: Psychology at the Human-Computer Interface, Cambridge: Cambridge University Press.
Craig, S.D., Gholson, B. & Driscoll, D. (2002). Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features and redundancy, Journal of Educational Psychology, 94 (2), pp. 428-434.
Harp, S. F. & Mayer, R. E. (1997). The role of interest in learning from scientific text and illustrations: On the distinction between emotional interest and cognitive interest, Journal of Educational Psychology, 89 (1), pp. 92-102.
Laurel, B. (Ed.). (1997). The Art of Human-Computer Interface Design, Reading, MA: Addison- Wesley.
Martinez, M. (2001). Key design considerations for personalized learning on the web, Educational Technology & Society, 4 (1), pp. 26-40.
Mayer, R. E. (2001). Multimedia Learning, Cambridge, UK: Cambridge University Press.
Mayer, R. E., Dow, G. T. & Mayer, S. (2003). Multimedia learning in an interactive self- explaining environment: What works in the design of agent-based microworlds?, Journal of Educational Psychology, 95 (4), pp. 806-813.
Mayer, R. E., Fennell, S., Farmer, L. & Campbell, J. (2004). A personalization effect in multimedia learning: Students learn better when words are in conversational style rather than formal style, Journal of Educational Psychology, 96 (2), pp. 389-395.
Mayer, R. E., Sobko, K., & Mautone, P. D. (2003). Social cues in multimedia learning: Role of speaker’s voice. Journal of Educational Psychology, 95 (2) ,419–425.
Moreno, R. & Mayer, R. E. (2000). Engaging students in active learning: The case for personalized multimedia messages. Journal of Educational Psychology, 93, pp. 724-733.
Moreno, R. & Mayer, R. E. (2004). Personalized messages that promote science learning in virtual environments, Journal of Educational Psychology, 96 (1), pp. 319-334.
Reeves, B. & Nass, C. (1996). The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. CSLI publications: California.
Schraw, G. (1998). Processing and recall differences among seductive details, Journal of Educational Psychology, 90 (1), pp. 3-12.
Spiro, R. J., Feltovich, P. J., Jacobson, M. J. & Coulson, R. L. (1992). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the Technology of Instruction: A Conversation, Hillsdale, NJ: Lawerence Erlbaum Associates.
Sweller, J. & Chandler, P. (1994). Why some material is difficult to learn, Cognition and instruction, 12, pp. 185-233.
Sweller, J. & Chandler, P. (1991). Cognitive load theory and the format of instruction, Cognition and Instruction, 8, pp. 293-332.
Toh, S. C. (2004). Reusable learning objects: vision and practice – a USM experience. In H.
Atan & R. M. Idrus (Eds.), Proceedings of 17th Educational Technology Convention, Tanjung Bungah, Pulau Pinang, pp. 319-331.
Tuovinen, J. E. (2001). Implications of discovery learning research for the design of flexible learning. In L. Richardson & J. Lidstone (Eds.), Flexible Learning for a Flexible Society:
Proceedings of ASET-HERDSA 2000 Conference. Toowoomba, Qld.
