This is to confirm that the work contained in this thesis entitled "User Interface Design Features and their effects on Computer Based Educational Assessment: A Cross-Cultural Study" submitted by Mr. 73 Figure 4.8 Instructional effectiveness of vertical menu-based navigation and step-based navigation CBA relationship between Indian and American students. 84 Figure 4.14 Instructional effectiveness of question matching and time matching CBA relationship between Indian and American students.
Computer based Assessment and Testing
- Introduction
- Objectives of the Research
- Rationale for the problem chosen to be investigated
- Computer- Based Assessment & definitions – setting frameworks
- History of Computer based Assessment
- Benefits of CBAs
- Types of CBAs
- Types of test administration
- Types of delivery of CBA systems
- Assessment content and CBA delivery systems
- Issues related to CBA systems
- Scope of the thesis
- Chapter Summaries
- Chapter 1: Summary
Assessment methods are crucial in meeting the objectives of the course, i.e. the expected study results. However, they all refer to the same concept of the electronic form of transmitting assessment content. They reported on cross-cultural data analysis and the impact of interface features on learners.
State of the Art Literature Review
Introduction
Previous studies on test mode effect
Factors affecting equivalence of test scores
- Scrolling
- Navigation
- Item presentation and review
- White space
Insight from state of the art review
- Research Questions
Chapter 2: Summary
Research methodology: Experiment, Hypothesis and Sample Design
Introduction
Research Questions
Aim of the study
Objectives of the study
Research Plan: Materials & Methods
- Posits under test in the study
- Working hypotheses
- Participants and Sample Design
- Instruments Used
Experimental processes followed
Experiment Design
- Independent Measures
- Dependent Measures
- GUI – Design of Experimental CBA Environment
Experiment Design – Rationale
Preparing for Data Analysis
Chapter 3: Summary
Results, Analysis and Discussions
Introduction
The research reported in this dissertation examines the effect of interface characteristics and cultural backgrounds of the students on their performance and their subjective evaluation of the assessment method in a CBA system. It is important to conduct such a study because the literature fails to address the issue of the impact of interface characteristics and cultural background on it. Although the current study does not examine the “test mode effect” per se, but the effect of different presentation formats within the same mode, here the computer-based mode of assessment delivery.
The effect of these interface features and cultural backgrounds was observed on the dependent measures, namely subjective cognitive load, perceived system usability, perceived ease of use, perceived usability, perceived system quality, relative advantage, willingness to continue using, perceived e-learner's satisfaction, total points scored in the test and time taken to complete the test. The present chapter presents the statistical analysis of the data collected during the five experimental studies. Working hypotheses of the five experiments were again stated and the results were not used in light of these hypotheses.
Results and Analysis
- Experiment 1: Scrolling vs. Without Scrolling
- Experiment 2: Vertical menu based vs. Step based navigation
- Experiment 3: Question Customization vs. Time Customization
- Experiment 4: Single question without customization vs. Question customization
- Experiment 5: Low Screen Density vs. High Screen Density
WCU Almost Normal Almost Normal Minor deviations Minor deviations PES Almost Normal Almost Normal Almost Normal Minor deviations TM Perfect Normal Perfect Normal Perfect Normal Perfect Normal. Another indication of the preference of American students for the role-based CBA condition can be observed from the instructional effectiveness plot (Figure 4.1). American students were found to complete both formats of the CBA at a much faster rate than their Indian counterparts.
RA Small deviations Almost Normal Almost Normal Small deviations WCU Small deviations Small deviations Small deviations Small deviations PES Small deviations Small deviations Almost Normal Almost Normal. TK Small deviations Almost Normal Small deviations Almost Normal perceived ease of use, perceived usefulness, system quality, e-learner satisfaction, willingness to continue use and relative advantage in a summative CBA environment. Univariate analysis performed to assess the effect of the criterion variables on the dependent measures reveals that adaptation characteristics influence subjective cognitive load, perceived system usability, perceived ease of use, perceived usability, perceived system quality, willingness to continue using, in total.
Q-Q plots indicated that there were slight deviations from normality for some of the dependent measures. Univariate statistics of dependent measures based on the independent factor - country and the interaction of factors. The results of the experiment emphasize that there is a significant interaction of interface functions and cultural background on the dependent measures of subjective cognitive load, perceived ease of use, perceived usefulness, relative advantage, perceived satisfaction of the e-learner and readiness for further use.
RA Almost normal Completely normal Completely normal Almost normal WCU Slight deviations Slight deviations Slight deviations Almost normal. A pairwise analysis conducted to determine the direction of effect indicates that American students reported significantly higher ratings of system usefulness for both CBA conditions compared to their Indian counterparts. Experiment results on the effect of low screen density (large white space) vs.
Chapter 4: Summary
Conclusion
Introduction
A review of the literature points to the existence of inconclusive research from the point of view of determining the effect of GUI on the individual characteristics of the subjects. Although it has been noted in the literature that individual characteristics may be a predictor of the test modality effect among the critical individual characteristics that may affect student performance in a computer-based assessment environment, they have not previously been found to be addressed. Furthermore, because computer delivery of assessment content allows test designers the flexibility to design innovative item types and presentation formats, it is argued that different presentation formats may affect student performance in a computer-based assessment environment.
Therefore, investigations have been carried out to optimize presentation formats that do not interfere with the individual characteristics of students and their performance. In other words, we compared between "Presentation Formats" (specifically the interactive components of GUIs) of CBAs to investigate and answer the question of whether Presentation Formats can be held responsible for variation in user performance in an exam - where content complexity is also tested since the user's intellectual capacity is assumed constant across different experimental conditions. Five experiments were conducted based on interface features such as scrolling, navigation mechanism, customization and screen density (white space).
Experts who had high computer attitude, low computer anxiety and had experience with CBA systems were allowed to participate in the main study. The Fulbright Foundation sponsored the studies conducted in the USA for which the researcher traveled to the USA to conduct the experiments in the American cultural environments. In each of the five experiments, interface characteristic and cultural backgrounds of the students were the independent measures.
Student performance in terms of test scores and time taken to complete the test was recorded.
Consolidated Findings
Their subjective experiences during the evaluation session were recorded in terms of subjective cognitive load, perceived system usability, perceived ease of use, perceived usability, perceived e-learner satisfaction, perceived system quality, relative preference, and willingness to continue using. The results of the research confirm the central argument of this thesis that special attention should be paid to the design of presentation formats in the CBA system, because if the students' performance were affected by this, the goal of assessment would be lost and students would only try to familiarize themselves with the presentation format, not to they impose their intellectual abilities to solve assessment problems.
Contribution of the thesis
- Theoretical Contribution
- Practical implications of the findings
- Summary of the contribution of the thesis
The current study suggests that students in computer-based assessment environment behave very differently than students in other components of hypermedia-based learning environment. The results of the research study highlight that students in CBA are not affected because of the navigation mechanisms, namely step or vertical menu. White space (Screen density): The published research literature has limited results and few studies investigated this feature.
Furthermore, they studied the effect of white space in comprehension-based test item layout, but almost no one reported on the effect of white space in MCQ (Multiple Choice Question) format and how graphical elements embedded in the GUI to help the students in a test format and thus reducing the white space can affect the students. The present study examines the issue of white space from this perspective and found that in contrast to the effect of white space in comprehension-based computer tests, in MCQ format-based computer tests students are not affected due to the reduction or increase in white space. Screen density does not appear to affect students in a KBA environment; thus, visual aids in the GUI can help students' performance.
These heuristics provide useful guidelines for GUI designers in their efforts to minimize the effect of interface features on student performance. The dissertation provides answers to the theoretical contradictions in the research literature and highlights the effects of interface features on student and other performance. An example of a KBA GUI design that incorporates the research results is illustrated in Figures 5.1 and 5.2.
This study provides the first cross-cultural analysis of the effect of presentation format on test performance and subjective evaluation of the CBA system by students from different cultural backgrounds (India and USA).
Inferences
This study also empirically investigated and documented the "presentation format effect" for the first time.
Limitations and future scope
Chapter 5: Summary
Hypermedia as an educational technology: A review of the quantitative research literature on learner comprehension, control, and style. Evaluating the effect of navigation tools on cognitive load in a computerized test format. The effect of violation of normality on the t test of the correlation coefficient. 2009).A case study of a computer-aided examination system.
Navigation in hyperspace: An evaluation of the effects of navigational tools and subject matter expertise on browsing and information retrieval in hypertext. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (pp. 191-200). An analysis of the learning responses of male and female students to different assessment environments.
The incomplete equivalence of the paper-and-pencil and computerized versions of the General Aptitude Test Battery. Read both statements and decide which of the following answer choices correctly depicts the relationship between these two statements. To determine which of them is the longest, which of the following further information, if any, is required.
If the dinner is given on the day the trees are watered, which of the following statements should be true. The police closed the entire area of the town for half the day this morning. A is sitting next to B, C is sitting next to D, D is not sitting with E who is on the left side of the bench.
