Our fifth recommendation, implementing pre and post surveys for PLO participants to
ascertain baseline data and long-term outcomes of the training intervention, aligns with research question #1 and finding #5. Parts of this recommendation are necessarily based on conjecture due to the scope of the capstone team’s charge. However, the finding that there is no process to mark participant’s baseline nor a follow-up process in place to measure the long-term outcomes of WIDA’s professional learning events is a significant one. This is because the perceived impact of training over time, level of learning and retention post-training, and application to participant’s teaching and learning contexts are all important factors that can help to assess the effectiveness of WIDA’s offerings. Baldwin and Ford’s Transfer of Training theory - which posits that the positive transfer of training is influenced by three factors: trainee characteristics, training design, and work environment - can help explain why these factors are important.
Trainee characteristics include the trainee’s motivation, prior knowledge, and ability to learn. Training design refers to the content of the training, the delivery method, and the opportunity to practice. Work environment includes supervisor support, the availability of adequate resources, and opportunities for application.
Drawing from this finding, the capstone team would suggest the implementation of pre and post-surveys - as found in Appendix B and Appendix C, respectively. The survey should be designed to capture data on the Baldwin and Ford model variables. This additional feedback may provide a valuable way to assess the effectiveness of the PLOs, can provide valuable insight into how to better meet the needs of educators, and can be used to improve the design and delivery of future PLOs.
Examples from a pre-training survey (for additional examples, see Appendix B):
● What is your experience with multilingual learners?
● How would you rate your ability to learn new skills?
● How supportive is your work environment of learning and development?
● What would help you feel adequately supported in your professional development?
● How confident are you that you’ll have the opportunity to use new learning in your work?
Examples from a post-training survey (for additional examples, see Appendix C):
● How relevant was the training to your work with multilingual learners?
● Did the training provide you with the opportunity to practice the new knowledge and skills?
● How well did the training address your individual needs as an educator of multilingual learners?
● What barriers have you experienced attempting to transfer the training to your work?
Discussion
Recommendations ultimately offered by the capstone team were developed in order to assist WIDA in assessing the perceived impact of its PLOs on K-12 educators for multilingual learners across its consortium and to identify the most useful feedback informing decision-making for future workshops. This supports the organization’s utmost mission to be the most trusted and valued resource in multilingual student language development. While complementing one another, our five recommendations differ in complexity and would require a range of efforts for proper implementation. For example, recommendation #2 suggests that survey participants should be able to view the survey in its entirety at one time. This could be a single-action, simple formatting change. In contrast, recommendation #5 suggests the design and dissemination of two entirely new surveys distributed pre and post workshop participation. These new surveys would not only need to be created, but would also require a systematic approach to releasing and following up with participants directly.
Moreover, no recommendation is reliant on another recommendation and the impact of each is unique. However, the team felt that recommendations #2 and #3 paired well together and could be addressed within the same time-frame. And, lastly, the capstone team believes finding #4 may be of most importance to the organization as it directly relates to the satisfaction of the two greatest populations of interest to WIDA (general education teachers and ESL/bilingual teachers).
The capstone team is confident that WIDA holds the most holistic knowledge of its own organizational resources, priorities and stakeholder needs. Therefore, WIDA should possess the autonomy to pick and choose the implementation and/or determine the order of recommendation implementation. Regardless of which recommendation is addressed, or in which order, the capstone team maintains that these recommendations will allow WIDA to gain a better understanding of its participants’ perceived impact and feelings towards PLOs and address areas of strength and areas of opportunity across its work in professional learning with K-12 educators around the country.
The capstone team offers several areas of consideration for future research. First, a longitudinal study of the impact of WIDA’s PLOs on K-12 educators and multilingual learners.
This study could track the progress of educators and learners over time to assess how their knowledge and skills change as a result of participating in WIDA’s PLOs. Second, an in-depth study of the factors found to influence the effectiveness of WIDA’s PLOs. This could help identify the most significant factors that contribute to the success of WIDA’s well-received PLOs, such as the quality of the content, the delivery method, and the support WIDA provides its participants.
Thirdly, WIDA could consider a study on the cost-effectiveness of WIDA’s PLOs. This study could compare the cost of WIDA’s PLOs with the benefits they provide, such as improved student achievement and increased educator confidence working with this special population of students.
A final interesting suggestion might be a study on the impact of WIDA’s PLOs on multilingual learners from different regions of the country or different cultural or linguistic identities. This study ties to much of what was discussed during our review of the literature. It could explore the variance of experiences of multilingual learners from differently identified languages and cultural backgrounds, with various funds of knowledge, and how WIDA’s PLOs can be tailored to meet the needs of the educators and of these learners.
Limitations
The capstone team was provided data from WIDA that consisted of 22,945 survey responses from a variety of PLO offerings from 2019-2022. In this data set, WIDA gave the capstone team access to three specific open-ended, written questions from the survey and some demographic information about its participants. However, the capstone team did not have access to the remainder of the survey questions or participants’ responses to these other portions.
Additionally, it is important to acknowledge that only a small sample of data (435/22,945 or 1.9%
of surveys) was ultimately analyzed. While this reduction was compared for overall representation with the initial dataset and performed intentionally in order to avoid issues of saturation, ensure a manageable scope of analysis for the capstone team, and to align with WIDA-identified parameters, the significant reduction in sample size is worth noting.
Another limitation includes the fact that the current survey distribution structure prevented the ability for the capstone team to follow up with participants. The capstone team decided the conceptual framework of Transfer of Training (Baldwin & Ford, 1988) was the best fit for analyzing the organization’s data to inform its ultimate areas of inquiry. However, WIDA did not design the initial survey with Transfer of Training in mind. Therefore, a slight disconnect existed because there was no current questioning in place in order to determine the nature of participants’
work environment or follow-up information collected to definitively ascertain whether application and retention of learning took place. For this reason, the capstone team could only identify named instances of ‘imagined transfer’ or participant’s named intentions to apply content learning into their classroom and workplace environment.
Lastly, aggregate survey data was collected from a range of PLOs but synthesized into a single analysis. This range included feedback from participants engaging in online, self-paced programs, online-facilitated webinars and all-day in-person learning opportunities. Regardless of the type of PLO, survey questions and responses were collected in the same manner. Additionally, the capstone team’s process of coding and interpreting data was consistent despite differences in the workshop delivery and structure.
Additional analysis of data could be completed with larger sample sizes or delineation of workshop type could be conducted in order to address the limitations of this study.
Conclusion
The capstone project was designed to provide a robust analysis of survey responses on WIDA’s PLOs. The goal of this investigation was two-pronged. Firstly, the capstone team aimed to assist the organization in determining the nature of the impact of its trainings. Secondly, it strove to provide useful data for future decision-making that could increase the effectiveness of WIDA’s work with educators and multilingual learners across the country. In this effort, the capstone team identified an applicable, well-established conceptual framework, distilled a final sample of analyzable survey responses, assessed and coded a robust set of qualitative data, discovered five key findings and developed five central recommendations to complete the investigation.
Through its professional learning offerings, WIDA engages in critical work supporting language acquisition for multilingual students and educators that strive to increase equitable
opportunities for their learning across the country. It also strongly ties to observed trends within the K-12 field of education, specifically, the increased presence of multilingual students in classrooms accompanied by a growing call for resources to better-serve these learners in the changing landscape of education. It is the hope of the capstone team that the information provided can be triangulated with previous WIDA findings in order to offer more holistic evidence on the impact of the organization’s pivotal work and equip WIDA with additional knowledge to make an even greater impact moving forward.
References
Aliakbari, M., & Malmir, B. (2017, February 28). Development and validation of an English language teacher learning scale. Cogent Education, 4: 1292613. Retrieved from https://www-proquest-
com.proxy.library.vanderbilt.edu/docview/1994426306/fulltextPDF/E5D31DD44ECA4 713PQ/1?accountid=14816
Alliger, G.M., et al. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-358.
Baldwin, T. &; Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41(1), 63–105.
Baldwin, T. T., Kevin Ford, J., & Blume, B. D. (2017). The state of transfer of training research: Moving toward more consumer-centric inquiry. Human Resource
Development Quarterly, 28(1), 17–28. https://doi.org/10.1002/hrdq.21278 Bandura, A. (1971) Social Learning Theory. General Learning Press, New York
Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341–347.
https://doi.org/10.1016/j.evalprogplan.2004.04.011
Board of Regents of the University of Wisconsin System. (2023). Mission and history. WIDA.
Retrieved from https://wida.wisc.edu/about/mission-history
Bramley, P. (1996). Evaluating Training Effectiveness (second edition), London: McGraw-Hill Chuckry, C. (2019). Removing Barriers to Profesional Learning. BU Journal of Graduate
Studies in Education, 11(2), 36-40.
Easterby-Smith, M. (1986). Evaluation of Management Education, Training, and Development,
Aldershot: Gower.
Callahan, R. M. (2013, February). The English Learner Dropout Dilemma: Multiple Risks and multiple resources. California Dropout Research Project - University of California Linguistic Minority Research Institute. https://cdrpsb.org/researchreport19.pdf Ferguson, R. (1991). Paying for Public Education: New Evidence on How and Why Money
Matters. Harvard Journal of Legislation, 28, 465-498.
Ford, J. K., Baldwin, T. T., & Prasad, J. (2018). Transfer of training: The known and the unknown. Annual Review of Organizational Psychology and Organizational Behavior, 5(1), 201–225. https://doi.org/10.1146/annurev-orgpsych-032117-104443
Freifeld, Lorri (Nov 16, 2022). 2022 Training Industry Report. https://trainingmag.com/2022- training-industry-report/
Gandera, P., Maxwell-Jolly, J., Driscoll, A. (2005). Listening to Teachers of English Learners.
Santa Cruz: Center for the Future of Teaching and Learning.
http://www.cftl.org/documents/2005/ listeningforweb.pdf
Goldstein, I. L. (1993). Training In Organizations: Needs Assessment, Development and Evaluation (third edition), Pacific Grove, CA: Brooks/Cole.
González, N., Moll, L., & Amanti, C. (2005). Funds of knowledge: Theorizing practices in households, communities and classrooms. Mahwah, NJ: Lawrence Erlbaum Associates.
Grapin, S.E., & Lee, (2021). WIDA English language development standards framework, 2020 edition: Key shifts and emerging tensions. TESOL Quarterly. Advance online
publication. 1-13. https://doi.org/10.1002/tesq.3092
Greeno, J & Gresalfi, M. (2008). Opportunities to learn in practice and identity. In P.A. Moss, D.C. Pullin, J.P. Gee, E.H. Haertel, & L.J. Young (Eds.), Assessment, equity and
opportunity to learn (pp. 170-199). New York: Cambridge University Press.
Grossman R, Salas E. The transfer of training: what really matters. International journal of training and development. 2011;15(2):103-120. doi:10.1111/j.1468-2419.2011.00373.x Guskey, T. R., & Suk Yoon, K. (2009, March). What Works in Professional Development?
The Leading Edge - Professional Learning. Retrieved February 13, 2023, from https://journals.sagepub.com/doi/pdf/10.1177/003172170909000709
Hill, H. C. (2009). Fixing Teacher Professional Development. Phi Delta Kappan 90(7), 470-476.
Holton, E. F. III. (1996). The flawed four-level evaluation model. Human Resource Development Quarterly, 7(1), 5–21. https://doi.org/10.1002/hrdq.3920070103
Hong, H. (2022). Maximizing learning opportunities with Young Refugee Multilingual Learners in urban schools: An asset-based participatory action research. Urban Education,
004208592210826. https://doi.org/10.1177/00420859221082668
Horn, M. B., & Goldstein, M. (2018, June 12). Putting school budgets in teachers’ hands. News - What’s Next - Vol. 18, No. 4. https://www.educationnext.org/putting-school-budgets- in-teachers-hands-end-users-classroom-purchasing-decisions/
Howell, E., Barlow, W., & Dyches, J. (2020, November 30). Disciplinary literacy: Successes and challenges of professional development. Journal of Language and Literacy Education.
Retrieved January 7, 2023, from https://eric.ed.gov/?id=EJ1300683
Katz, S. & Dack, L. (2013). Intentional Interruption: Breaking down learning barriers to transform professional practice. Thousand Oaks, CA: Corwin Press.
Keep, E. (1989). Corporate training: the vital component? in New Perspectives on Human Resource Management. J. Storey (ed). London: Routledge.
Kirkpatrick, Donald (1967) Evaluation of training. Training and development handbook (86-
112). New York: McGraw-Hill
Kirkpatrick, D. (1996). Great ideas revisited. Training and Development, 50(1),54-65.
Knight, J. (2011). Unmistakable Impact: A Partnership Approach for Dramatically Improving Instruction. Corwin Press.
Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press.
Luthans, F. (1989) Organizational Behavior. 5th Edition, McGraw-Hill Inc., New York.
McDonald, L. (2011). Transfer of training in teacher Pd: A process-outcome orientation.
Procedia - Social and Behavioral Sciences, 29, 1885–1894.
https://doi.org/10.1016/j.sbspro.2011.11.438
National Center for Education Statistics. (2022, May). The Condition of Education: English Language Learners in Public Schools.
https://nces.ed.gov/programs/coe/indicator_cgf.asp.
NYU Steinhardt . (2018, May 1). 1 in 4 Students is an English Language Learner: Are We Leaving Them Behind? Counseling@NYU Blog.
https://counseling.steinhardt.nyu.edu/blog/english-language-learners/.
Pınar, S. & Bardakçı M. & Arslan Yalçın F. (2021). Factors Influencing Teachers’ Professional Learning: A Study of Turkish EFL Teachers . Journal of Language and Linguistic Studies, 17(Special Issue 1), 173-192.
https://www.researchgate.net/publication/349532843_Factors_Influencing_Teachers%2 7_Professional_Learning_A_Study_of_Turkish_EFL_Teachers
Popova, A., Evans, D., Breeding, M., Arancibia, V. (2022). Teacher Profesional Development around the World: The Gap between Evidence and Practice. The World Bank Research
Observer, 37(1). Oxford University Press.
Prater, M. & Devereaux, T. (2009). Culturally Responsive Training of Teacher Educators.
Action in Teacher Education, 31(3), 19–27.
https://academia.edu/1895091/Culturally_Responsive_Training_of_Teacher_Educators Reio, Thomas & Rocco, Tonette & Smith, Douglas & Chang, Elegance. (2017). A Critique of
Kirkpatrick's Evaluation Model. New Horizons in Adult Education and Human Resource Development. 29. 35-53. 10.1002/nha3.20178.
Samson, J. & Collins, B. (2012). Preparing All Teachers to Meet the Needs of English Language Learners: Applying Research to Policy and Practice for Teacher Effectiveness. American Progress. https://files.eric.ed.gov/fulltext/ED535608.pdf.
Santos, A. & Stuart, M. (2003). Employee perceptions and their influence on training effectiveness. Human Resource Management Journal, 13(1), 27-45.
https://doi.org/10.1111/j.1748-8583.2003.tb00082.x
Soland, J. & Sandilos, L. (2020). English Language Learners, Self-efficacy, and the Achievement Gap: Understanding the Relationship between Academic and Social- Emotional Growth. Journal of Education for Students Placed at Risk (JESPAR) 26(4), 1- 25.
Sparr JL, Knipfer K, Willems F. How Leaders Can Get the Most Out of Formal Training: The Significance of Feedback‐Seeking and Reflection as Informal Learning Behaviors.
Human resource development quarterly. 2017;28(1):29-54. doi:10.1002/hrdq.21263 Stevens, F.I., & Grymes, J.A. (1993). Opportunity To Learn: Issues of Equity for Poor and
Minority Students.
Subedi, B.S. (2004) Emerging Trends in Research on Transfer of Learning. International
Education Journal 5(4).
U.S. Department of Education . (2022, October 13). U.S. Department of Education Awards nearly $120 million over five years to support educators of English learner students.
Archived Information. https://www.ed.gov/news/press-releases/us-department-
education-awards-nearly-120-million-over-five-years-support-educators-english-learner- students
War, P., Bird, M., and Rackham, N. (1976). Evaluation of Management Training, London:
Gower Press
WIDA, University of Wisconsin-Madison. (2023). Mission and history. WIDA.
https://wida.wisc.edu/about/mission-history
Yoon, K., Duncan, T., Wen-Yu, S., Scarloss, B., Shapley, K. (2007). Reviewing the Evidence on How Teacher Professional Development Affects Student Achievement. Issues &
Answers. REL 2007-No. 033, Regional Educational Laboratory Southwest.
Appendix A
Table 1: Codebook
Category Code Description of Code Highlights From Coding Trainee Characteristics Teacher Motivation ● Openness to
collaborate
● Openness to engage
● Intention to transfer
● Intention to implement
“I learned a lot in a short period of time. I will definitely use this
information to better my walkthroughs as an administrator and share information with colleagues and
educators under my supervision.”
(Administrator #1)
“The most worthwhile thing about this workshop was learning practical skills that I can apply
immediately in the classroom with my newcomers… I learned so much in my course on how to address the needs of newcomers in
the classroom. I felt that all the information was highly relevant
for my current position. I was able to start implementing some
of the strategies mentioned before even completing the course. I was really excited to begin trying new things in the classroom and to see how well my newcomers responded to the
support. I am really thankful for this training and look forward to
future workshops.” (Gen Ed.
Teacher #105)
Trainee Characteristics Ability “Thank you for this free response.
I know I should have my ducks in a row, but as a new to ELL education teacher who believes in
and wants to be doing this important work, I'm overwhelmed
(appropriately so) by all that should happen in the best- practices ELL program. I am committed to do those things, but I lack some optimism about
my ability to master these visions and strategies fast enough and effectively enough to
advocate for and assist others in wanting to join the effective fun.
In other words, I need some time management skills and some time.
Can you give that?”
(ESL/Bilingual Teacher #22)
“This e-workshop was incredibly helpful for all levels of introductory background with the
content. I had advanced knowledge and thought the entire course was fantastic and supportive with resources, ideas, examples, use of technology and format. I am recommending this course to every teacher in my school. Any school that has 1 newcomer/ EL student should in
fact take this course.”
(ESL/Bilingual Teacher #432) Trainee Characteristics Reflection ● Shifts in mindset
● Deepening understanding
● Imagining concept (not mentioning specific intention to implement but considering concepts in context)
“This course provides sample walkthroughs and tools for leaders
to support multilingual learners in the classroom. It is helpful to see the WIDA synthesized research-
based practices in education standards and I now have a better
understanding of how to apply these strategies in my teaching
career.”
(General Ed. Teacher, #119)
Training Design Credibility “WIDA is constantly redefining
itself within the spectrum of education, academia and professional development with a
primary concern for student centered development. Any one training has been beneficial for me
in better understanding who I am providing service to and what ways I can improve that service.”
(ESL/Bilingual Teacher #107)
“Engaging Multilingual Newcomers eWorkshop was the
reinforcement of concepts I learned while a student for my
Master's in TESOL. Be[ing]
reminded of the contributions of academics like Swain, Vygotsky,
and Krashen among many others was valuable. Also, re- immersing myself in the world of
scaffolding and differentiation based on approximate student needs is always a good thing.”
(ESL/Bilingual Teacher #107) Training Design Demonstration ● Modeling
● Scenarios
“I especially enjoyed the lesson on FRACTIONS at the SMALL GROUP table. It was great to see
this being modeled by an effective teacher”
(ESL/Bilingual Teacher #31)
“I enjoyed the training. It was a good mix of instruction and
interactive components. I especially liked seeing real-
world examples that demonstrated how to use the
WIDA framework to plan content-area instruction that supports language development
for all learners.”
(General Ed. Teacher, # 99) Training Design Differentiation ● Level of difficulty
● Variety of contexts
● Assumed background knowledge
“This was probably the easiest workshop for me out of the ones I
have taken. I taught Social Studies for years so this course