The LiteracyAccess Online (LAO) project1 provides an example of an integrative learning design study based on the ILDF specifically illustrating the intersection and systematic expression of multiple design and research methods. LiteracyAccess Online is an effort to utilize Web-based technology to provide support for teachers, tutors, and parents (literacy facilitators) in addressing literacy goals for all children with a particular focus on those with disabilities. After four years of design research and development, LAO (http://
literacyaccessonline.com) now provides a technology-based learning environment that promotes the use of specific literacy strategies for the improvement of tutoring and reading performance as the child and literacy facilitator collaboratively engage in the process of reading online.
The Informed Exploration Phase
The exploratory research objectives of the LAO integrative learning design study were two- fold;
1) to investigate the nature and effectiveness of a consistent technology-based, collaborative literacy environment as well as;
2) to generate knowledge about how literacy facilitators and children understand and employ reading support strategies.
These objectives were originally conceived as research/evaluation questions and evolved from an extended, progressive investigation into the provision of literacy support for facilitators and children. This “meta-methodological” design research process consisting of multiple research methods (e.g. survey, focus groups, interviews, expert reviews, etc) resulted in clearly articulated learning targets, task analyses of learning objectives, theoretical model embedded in a technology system design and congruent research/
evaluation questions that drove more rigorous qualitative testing of the intervention whose results contributed to theory of literacy support for children with disabilities further elaborated in sections below.
To begin exploratory research cycles, initial explorations into target audience and
stakeholder perceptions, related products and literature and documentation of the complex nature of supporting literacy revealed many plausible paths for design research. The interdisciplinary research team involved in the LAO project were charged with determining the research direction and consisted of educational researchers, teachers, graduate students, content experts in literacy, special education and assistive technology as well as parents involved in an advocacy group for children with disabilities. The broad design research
1) The LiteracyAccess Online (LAO) project is supported by the Office of Special Education Programs in the Department of Education Steppingstones of Technology Innovation for Students with Disabilities Grant CFDA84.327A
focus evolved from the team’s perceived lack of support for children who were struggling with the literacy process, based on direct observations of this problem in both classroom and home environments which then manifested itself into several individual but connected research studies. For example, we conducted multiple interviews with parents with children with disabilities who were struggling with the reading process. We also invited several parents to participate on our design research team. The teams’ analysis of the interview data as well as the design research team discussions revealed our initial approach. We had initially decided to design a tutorial-based intervention only for the child’s benefit, however, a comment in a team meeting dramatically changed our design direction. In line with Collins’ notion of conscious considerations of design as
demonstrating core underlying design decision-making, one parent member/stakeholder on the team stated that she primarily read in conjunction with (not to) her son and wanted to do so online but with additional support of higher level reading strategies. Based on that input and follow-up micro-cycles of interview data collection and analysis to confirmation the viability of this design approach with parents, we conducted a series of interviews and surveys to determine the feasibility of this design direction. The results of our investigation and discussions evolved into an online collaborative performance support system to support both literacy facilitators and their children in the literacy process as the determined design direction. This was a design decision based on data analyzed from multiple interviews and survey procedures. Aligned with Confrey and Lachance’s (2000) notion of drawing key inferences from dissatisfaction with current educational practices and direct experiences with children, initial theoretical conjectures were developed based on the analyzed data that advocated for reading, writing and assistive technology support for children with or without disabilities to increase their engagement and performance in literacy.
While these initial theoretical conjectures provided a central premise and broad direction for design research, more information was needed to refine these conjectures resulting in a comprehensive needs analysis and literature review that provided a firm and
complementary theoretical foundation for the intended design. Extensive exploration into appropriate literacy strategies, tutorial programs and processes, surveys of experts, teachers and parents as well as qualitative observation of children and facilitators engaged in a literacy experience all informed this phase of the research. This provided not only well- defined design directions but also added to the research literature regarding children with disabilities and their parents understanding of assistive technology and literacy learning (see Jeffs, Behrman & Bannan-Ritland, 2006). Many potential design research directions were considered based on the initial conjectures, however, data drawn from conducted interviews, direct experience with potential research participants and literature review converged and pointed the team in a particular direction.
A prominent theme that emerged across initial interviews, surveys and observations with experts, parents, teachers and children revealed that literacy facilitators had a crucial role in providing support for children struggling to gain literacy skills and the question remained how to best support this role. These findings and related literature provided insight for informed theory and improved conjectures based on the aforementioned qualitative interviews and literature reviews. Results from data collection and literature review methods in the informed exploration phase indicated that:
1) children can, but often do not use effective metacognitive reading strategies;
2) explicitly teaching these strategies can greatly enhance children’s comprehension of text;
3) teachers (as well as other literacy facilitators) need to be trained in how to provide cognitive structure for their students so that children can learn to guide their own generative processes in reading; and
4) one-to-one tutoring is one of the most effective forms of instruction for improving reading achievement but increased success often depends upon the skill of the tutor or facilitator and the establishment of consistent roles and expectations (Wittrock, 1998;
Wasik, 1998).
This exploration into the literature and perspectives of those involved in these issues greatly refined our initial theoretical conjectures and resulted in a dramatic change of our intended design direction for this research from a didactic, tutorial, child-focused
intervention to a collaborative, story-based reading experience providing embedded metacognitive strategy support for both the literacy facilitator and the child’s use. The rationale for this research direction was documented in a comprehensive needs analysis that detailed the data collection, conclusions and related literature review.
The next stage of our design research involved the analysis and description of the range of learners and facilitators that would potentially use the LAO system. Direct experience with 4th-8th grade children with or without disabilities, teachers, tutors, and parents provided data that characterized our audience. These descriptions were depicted as role models (Constantine & Lockwood, 1999) or personas (Cooper, 1999) that comprised abstract composite profiles of audience characteristics gleaned from actual interviews and observations and provided a focal point for design. Role models or personas are similar to Graue and Walsh’s (1998) qualitative vignettes that strive to capture the substance of a setting, person or event to communicate a central theme of qualitative data, based on multiple direct observations and are employed here as also a focal point for design.
Exploring the nature of the identified educational problem, related products and literature as well as creating and refining theoretical conjectures and descriptions of the audience provided an informed perspective for grounded design of a learning environment based on
articulated theory. These activities resulted in specific research artifacts including a needs analysis that contained an extensive literature review, an articulated and congruent design and research direction and detailed audience analysis based on qualitative and quantitative data. These documents were housed on a project Web site that provided a communication mechanism between team members as well as an archive of shareable design research processes, products and evidentiary data.
The Enactment Phase
The embodiment of the results of our informed exploration and theories about providing literacy support for children and literacy facilitators in a usable learning environment were collaboratively constructed across several stages and constructive research cycles that develop solutions to problems culminating in a Web-based prototype. The initial design of the LAO learning environment resulted directly from the design implications articulated in the previous phase of exploratory research, analyses and review. These implications were translated into an articulated prototype initially developed by building an abstract, paper- based model of the system for researcher and teacher input according to procedures adapted from usage-centered design processes previously mentioned as role models (Constantine & Lockwood, 1999). Role models are a technique to characterize primary and secondary target audiences for the purposes of design. For example, we created role models and personas for children with learning disabilities (such as attention deficit disorder) based on our direct experience with a child who was struggling in the reading process and his mother who did not have any knowledge of advanced reading support strategies. These techniques are based on real-world experiences with representatives of the target
audiences your intervention or system is being designed for but evolve into a archetypal composite of the attributes of many individuals. Therefore, role models and persona’s become a qualitative profile to continually target design efforts to maintain the audience(s) or user(s) perspectives.
Abstract or low-fidelity modeling/prototyping of the instantiated or enacted design provided opportunities for input and co-construction of LAO with several audience members prior to the more time-intensive computer-based production of the learning environment. We utilized Constantine and Lockwood’s (see foruse.com) procedures of usage-centered design that encompassed low-fidelity representation and organization of all the features of the database-driven Website. For LAO, we deliberately ultimately designed a Web database system that would permit performance support for the parent- child dyad in providing meta-cognitive prompts for both participants based on research- based reading strategies throughout a collaborative and generative process of engaging with text.
In the context of a constructive research approach that attempts to validate a particular construct (e.g. theory, model, software or framework) against identified criteria or benchmarks, the team conducted several iterative cycles of data-gathering and analysis of expert reviews and target audience reviews. These progressive, micro-cycles of data
collection and analysis resulted in data-driven cyclical revisions of the articulated prototype which were reflected in detailed design documentation including the production of
flowcharts, technical specifications and storyboards. The design research process of employing micro-cycles of constructive research data gathering and analysis elicited feedback at each cycle and design revisions agreed upon by the team which resulted in the initial creation and then progressive improvement of a Web-based prototype validated by data collected in a constructive research approach. As a team, we constructed specific criteria related to usability of system and observations and video analysis of actual use of the system by literacy facilitators and children. The specific methods of data collection employed at this stage included designer logs posted on the project Web site, expert panel reviews of the design and documented reviews of the design by content experts, audience members and the research team.
The Local Impact Phase
Once a physical Web-based prototype was in place, the incorporation of formative
evaluation and qualitative methods in an empirical manner could commence and began to characterize the rich, highly iterative nature of the local impact phase as it progressively informed, revised and refined our theoretical constructs as well as the Web-based instructional design approach and redesign efforts. The complex interactions between facilitators and children that can occur in multiple settings formed the series of micro- cycles in LAO examining these specific constructs that grounded related research questions:
1) parent-child dyads in an informal setting with extensive involvement by researchers; 2) parent-child dyads in a structured workshop experience supported by researchers and; 3) pre-service-teacher dyads in a field trial progressing toward more closely modeling authentic conditions experienced with the prototype. When a fully functioning prototype was not yet available, studies attempted to closely mimic the tasks that would be
embedded in LAO. The data gathering across these three studies incorporated observations, interviews, child and parent journal entries, videotaped use of system and pre- and post-online surveys (see Jeffs, Behrmann & Bannan-Ritland, 2006). This multi-tiered, multi-method evaluation scheme generated useful knowledge and subsequent results from each stage of inquiry were then cycled into changes of our theoretical conjectures, research design as well as system design. This process revealed insights into the core design principles (van den Akker, et al., 2006) that may support the collaborative learning and implementation of metacognitive processes by literacy facilitators and children in a technology-based environment.
Our core design principles that evolved and were refined included the following that when engaged in a collaborative literacy process that provides high level reading strategy metacognitive support in a Web-based context:
1) parent literacy facilitators could develop greater awareness and skill in implementing reading activities and identify supports for their child in a structured setting;
2) children showed improvement in literacy skills using technology-based support when participating in a guided workshop environment; and
3) pre-service teachers felt that the strategies and activities embedded in the LAO environment facilitated children’s comprehension, motivation and interest when working with them in this environment (see Jeffs, et. al. 2006).
More rigorous evaluations are planned for the future to systematically increasing number of participants and varying contexts for the use of LAO in school, home and tutoring environments. These studies involve detailed tracking of computer-based activities of the dyads in school and home settings, assessment of facilitator and child use of metacognitive strategies prior to using LAO and pre- and post comprehension measures after several weeks of using the system.
In addition, a series of expert reviews, usability testing, one-to-to-one, small group, and field testing were implemented in progressively more authentic settings. The more intensive studies focused primarily on qualitative studies that characterized the target audience interaction with the enacted theoretical model in relation to the learning targets.
Specifically, the team was interested in how facilitators and learners perceived and interacted with Web-based support in the collaborative literacy process (which included both reading and writing tasks). A pilot study was initially conducted that simulated some tasks within LAO and provided feedback on the emerging site with five dyads of mostly parent facilitators and one sibling facilitator. Methods included collecting data through semi-structured interviews and observations of parent-child interaction with the prototype and complementary assistive technologies (e.g. text-to-speech, etc.) that promoted in reading and writing activities.
The preliminary study revealed that the children were motivated to complete reading and writing activities on the Web and that facilitators developed awareness for implementing reading activities in a collaborative process but desired additional support for children’s disabilities. While the Web-based activities and supports for the reading process were useful for providing more authentic and self-initiated reading and writing activities, the research also revealed that interaction between parent and child dyads during these activities often created tensions that were not present when children were working with non-family members. Revisions to the theoretical model and enacted design of LAO based
on this cycle of evaluation included among others, behavioral prompts directed toward the parent-child dyad to potentially release tension (such as prompts to take a break, positive reinforcement techniques, etc.) when engaged in collaborative reading and writing tasks and additional reading strategy supports and activities.
To further investigate the enacted theoretical model, a follow-up small group qualitative study was conducted with eight parent/child dyads that represented a variety of skill levels and disabilities (Jeffs, 2000). The specific goals of this cycle of research was to identify the characteristics of parent/child dyads working together specifically in literacy skill development, depict the interactions of the dyad and investigate the impact of various forms of technology (Internet, EPSS and any assistive technology) on attitudes of the participants. Participants included parents and children with various disabilities in grades 4th through 6th who were reading at least two grades below grade level and had a tendency to avoid reading and writing tasks prior to participation in the study. The study revealed that parents recognized the importance of immediate feedback and assistive technology features in the provided tools. Other results revealed that with the support of their parents, children can select appropriate technologies and with integrated use of the Internet and assistive technologies, children’s writing samples improved in both quantity and quality. Suggested revisions for the LAO prototype based on these results included built-in assistive technology features (instead of merely references to outside resources) such as text-to-speech capabilities and reading selections reflecting varying abilities and areas of interest – features that were subsequently incorporated into the LAO design.
In each of these cycles of problem-state, data collection, analysis and subsequent design move or formative evaluation process, the theoretical model enacted within the LAO prototype expanded to incorporate new and revised elements based on targeted data collection and research results. At this point, traditional research and design processes somewhat diverge in that the analyzed results are not an end in and of themselves, but are used for data-driven decision making or problem solving to build upon or revise theoretical assumptions and improve design. Often, based on testing results, we would need to throw out previous prototype features and totally redesign, revise or add new features. The team’s informed design judgment and collaborative social negotiation was key to this decision- making.
The local impact phase is a time-intensive phase with multiple cycles that strives to yield a usable and internally valid intervention. Testing the intervention in progressively more realistic settings provides valuable information to inform theoretical assumptions related to the design but also to begin to isolate variables that might be further empirically tested.
In the LAO research conducted to date, the integration of reading strategy scaffolds and
assistive technology supports in the collaborative literacy process between facilitators and children with a range of disabilities was identified as one factor, of many, that seem to hold promise for improving literacy skills. Conducting additional research to further investigate the collaborative process promoted by the technological environment as well as isolating the effects of the multiple reading supports and assistive technologies afforded by the prototype remains an important objective in this research.
Although the funding cycle for LAO has ceased, in order to progress from local effects to more externally generalizable effects, additional cycles of testing are needed to isolate and test particular variables using multiple sites, diverse participants and settings progressively limiting the researcher-participant interaction. Based on available funding, field tests or trials are planned for LAO to collect significant amounts of quantitative and qualitative data from several sites and over 50 participant dyads using selected measurements, online surveys and interviews including parents and children in home school environments, pre- service teachers and in-service teachers that could represent other literacy facilitators in several geographical locations interacting with children with a range of disabilities. This data would provide additional evidence for the effectiveness of enacted theoretical assumptions for the collaborative reading and literacy process as well as provide evidence for the effectiveness of the prototype at its highest fidelity in full context of the intended use.
The Broad Evaluation Phase
The last phase of this design-based research effort involves disseminating LAO into the broad educational system. Although the LAO research has not yet fully progressed through this stage, initial explorations in this area have yielded some unique insights into the dissemination process. However, the reader should note that the dissemination process can encompass an entire research effort in itself. For example, Fishman (2006) has applied a design research framework related to the sustainability of technology-based curriculum interventions within an entire school district or system.
LAO, as a Web-based learning environment, affords the opportunity to publish current working prototypes online for open use and input that has resulted in an early and unique diffusion and adoption process begun prior to the completion of a fully functioning system.
While still in development, we have tracked over 100 potential adopters that have discovered and explored the LAO site. The profiling and data-base capabilities of the site permit tracking and analysis of this information that has provided detailed information on potential adopters of the system providing significant insight and impact on sources for our later diffusion efforts. We plan to incorporate more sophisticated computer-based data collection and analysis techniques such as datamining (Tsantis & Castellani, 2001) that may yield even more insights into early adopters’ behaviors, profiles and use of this new tool. We