Introduction
The purpose of this research project was to identify the exemplars and best practices of spiritual formation in online undergraduate programs among CCCU member institutions. Prior to this research, no consensus existed with regard to these best practices;
therefore, further research was needed to establish them. Since consensus among expert opinion was sought, the Delphi method served as the research methodology for this project.
This method seeks to establish consensus among a panel of experts through an iterative survey process where no consensus previously existed. When there “is little previous research” and “there is some benefit of achieving consensus,” the Delphi approach serves as the “the method of choice.”1 The value of the Delphi approach is in “provid[ing]
important insights from a panel of expert educators when developing standards, guidelines, and determining future trends.”2
The following questions guided this research project:
1. What are the best practices implemented by CCCU member institutions to provide effective spiritual formation in their online undergraduate programs?
2. What can be learned from those exemplars and best practices?
3. What are specific practices that administration and faculty can implement to be intentional about spiritual formation in an online environment?
1Wesley Vernon, “The Delphi Technique: A Review,” International Journal of Therapy &
Rehabilitation 16, no. 2 (2009): 71.
2Ravonne A. Green, “The Delphi Technique in Educational Research,” SAGE Open 4, no. 2 (April 2014): 2.
Design Overview
The Delphi Method was first introduced in the early 1950s by researchers Norman Dalkey and Olaf Helmer of the RAND Corporation.3 Their aim was to establish consensus and develop forecasting models with regard to various technological
advancements and military defense projects. One of the most prominent projects where Dalkey and Helmer utilized the Delphi method was in an attempt to find reliable expert consensus on the “number of [atomic] bombs required to reduce the munitions output” of a designated country.4
The purpose of the Delphi method is to establish consensus among experts with regard to a given problem. According to Skulmoski, Hartman, and Krahn, the Delphi method is “an iterative process to collect and distill the anonymous” input of experts related to a prescribed problem.5 Ravonne Green defines the Delphi Method as a method for forecasting “future events and for generating a quick consensus by a group” that collects data through anonymous surveys as a means to “quantifying group judgment.”6 As previously stated, when there “is little previous research” and “some benefit of achieving consensus,” the Delphi approach serves as the “the method of choice.”7 The value of the Delphi approach is in identifying “important insights from a panel of expert educators” in order to establish “standards” and best practices.8
3Harold A. Linstone and Murray Turoff, eds., Delphi Method: Techniques and Applications (Reading, MA: Addison-Wesley Educational, 1975), 10.
4Norman Dalkey and Olaf Helmer, “An Experimental Application of the DELPHI Method to the Use of Experts,” Management Science 9, no. 3 (April 1963): 1.
5Gregory Skulmoski, Francis Hartman, and Jennifer Krahn, “The Delphi Method for Graduate Research,” Journal of Information Technology Education: Research 6, no. 1 (January 2007): 1.
6Green, “The Delphi Technique,” 2.
7Vernon, “The Delphi Technique,” 71.
8Green, “The Delphi Technique,” 2.
Because the Delphi method uses a panel of experts to collect qualitative data and also statistically analyzes their responses to quantify that data, it can be viewed and used as a qualitative, quantitative, or mixed methods approach. Experts in research design, John W. Creswell and Vicki L. Plano Clark suggest that a mixed-methods design includes the following essential characteristics where the researcher
1. collects and analyzes persuasively and rigorously both qualitative and quantitative data (based on research questions);
2. mixes (or integrates or links) the two forms of data concurrently by combining them (or merging them), sequentially by having one build on the other, or embedding one within the other;
3. gives priority to one or to both forms of data (in terms of what the research emphasizes);
4. uses these procedures in a single study or in multiple phases of a program of study;
5. frames these procedures within philosophical worldviews and theoretical lenses;
6. and combines the procedures into specific research designs that direct the plan for conducting the study.9
They further qualify that an exploratory sequential design is a “two-phase sequential design that . . . starts by qualitatively exploring a topic before building to a second, quantitative phase” making the research “iterative” in nature.10 This project was therefore a mixed-methods exploratory sequential design where both qualitative and quantitative data was collected and analyzed within iterative phases that build on one another utilizing the classical Delphi approach.
The Delphi process. The Delphi Method is rooted in a research question and, as Dalkey and Helmer note, it is “centered around some central problem” in need of a consensus among experts.11 This method seeks to establish consensus among experts regarding solutions, forecasts, or recommendations related to the research question. The
9John W. Creswell and Vicki L. Plano Clark, Designing and Conducting Mixed Methods Research, 2nd ed. (Los Angeles: Sage, 2010), 5.
10Ibid., 86.
11Dalkey and Helmer, “An Experimental Application,” 2.
process involves developing a panel of experts and then soliciting their input through iterative rounds of anonymous interview questions. Dalkey and Helmer suggest,
The questions . . . are designed to bring out the respondent’s reasoning that went into his reply to the primary question, the factors he considers relevant to the problem, his own estimate of these factors, and information as to the kind of data that he feels would enable him to arrive at a better appraisal of these factors and, thereby at a more confident answer to the primary question.12
Skulmoski, Hartman, and Krahn suggest that there are four essential features of a Delphi study:
1. Anonymity of Delphi participants: allows the participants to freely express their opinions without undue social pressures to conform from others in the group. Decisions are evaluated on their merit, rather than who has proposed the idea.13
2. Iteration: allows the participants to refine their views in light of the progress of the group’s work from round to round.14
3. Controlled feedback: informs the participants of the other participant’s
perspectives, and provides the opportunity for Delphi participants to clarify or change their views.15
4. Statistical aggregation of group response: allows for a quantitative analysis and interpretation of data.16
By maintaining anonymity and controlling the interaction of the experts, the Delphi method avoids possible information bias, confrontation of opposing views, defensiveness, dominant personalities, and “groupthink.”17 Iteration and controlled feedback allows individual experts on the panel to see the “collective group response” and gives an
12Dalkey and Helmer, “An Experimental Application,” 2.
13Skulmoski, Hartman, and Krahn, “The Delphi Method,” 2.
14Ibid.
15Ibid., 3.
16Ibid.
17Dalkey and Helmer, “An Experimental Application,” 2; Chia-Chien Hsu and Brian A.
Sandford, “The Delphi Technique: Making Sense of Consensus,” Practical Assessment, Research &
Evaluation 12, no. 10 (January 2012): 2; G. David Garson, The Delphi Method in Quantitative Research (Asheboro, NC: Statistical Associates Publishers, 2013), loc. 144, Kindle.
opportunity for panelists to revise their own answers.18 This process “appears to be more conducive to independent thought on the part of the experts and to aid them in gradual formation of a considered opinion,” according to Dalkey and Helmer.19 The process ends when consensus is reached.
In addition to the essential features suggested by Skulmoski, Hartman, and Krahn, there are a number of key steps in successfully implementing the Delphi Method:
1. Developing the Research Question 2. Developing the Delphi Panel 3. Developing the Round 1 Survey
4. Evaluating & Analyzing Round 1 Results
Participants Given Opportunity to Revise Round 1 Answers after Reviewing Panel Responses
5. Develop Round 2 Survey Based on Round 1 Results 6. Evaluate & Analyze Round 2 Results
Participants Given Opportunity to Revise Round 2 Answers 7. Develop Round 3 Survey Based on Round 2 Results
8. Process Ends When Consensus Is Reached 9. Analyze and Document Results20
Therefore, Skulmoski, Hartman, and Krahn state that with a Delphi study, “the sample size varies . . . from 4 to 171 ‘experts.’”21 The panel that emerged from initial research of all CCCU faculty and administrators teaching courses offered and/or required
18Vernon, “The Delphi Technique,” 71; Hsu and Sandford, “The Delphi Technique,” 2.
19Dalkey and Helmer, “An Experimental Application,” 2.
20These steps were gathered from Green, “The Delphi Technique,” 3; Hsu and Sandford, “The Delphi Technique,” 2; Skulmoski, Hartman, and Krahn, “The Delphi Method,” 4; Vernon, “The Delphi Technique,” 71.
21Skulmoski, Hartman, and Krahn, “The Delphi Method,” 5.
within Christian ministry departments that incorporate intentional spiritual formation efforts served as the Delphi panel for this research project.
The Delphi panel. In a Delphi study, the participating panel of experts is critical.22 Since one is after expert opinion in a Delphi study, Skulmoski, Hartman, and Krahn suggest, “A purposive sample is necessary where people are selected not to represent the general population, rather their expert ability to answer the research
questions.”23 In fact, according to Garson, 95 percent of Delphi panels are selected using
“purposive rather than random sampling.”24 As mentioned in chapter 1, criteria for
“expertise” is defined, according to Skulmoski, Hartman, and Krahn, as “knowledge and experience, willingness to participate, sufficient time to participate, and effective
communication skills.”25 Further qualifications for expertise are typically outlined in each respective study. Garson suggests that research shows an “optimal” panel size to be between 11 and 20 experts.26 The size of the sample is affected by whether the panel is heterogeneous or homogeneous. Skulmoski, Hartman, and Krahn suggest, “Where the group is homogeneous, then a smaller sample of between ten to fifteen people may yield sufficient results.”27 Since this current study is not seeking to establish consensus among diverse opinions, but rather like-minded evangelical Christian educators among CCCU
22Skulmoski, Hartman, and Krahn, “The Delphi Method,” 3.
23Ibid., 4. Linstone and Turoff suggest that traditional Delphis “were characterized by a strong emphasis on the use of consensus by a group of experts’ as the means to converge on a single model or position on some issue.” Linstone and Turoff, Delphi Method, 29.
24Garson, The Delphi Method, loc. 495.
25Skulmoski, Hartman, and Krahn, “The Delphi Method,” 4.
26Garson, The Delphi Method, loc. 495. As noted in chap. 1, since expertise is the criterion of the Delphi Panel, Skulmoski, Hartman, and Krahn assert, “The sample size varies . . . from 4 to 171
‘experts.’” Skulmoski, Hartman, and Krahn, “The Delphi Method,” 5.
27Skulmoski, Hartman, and Krahn, “The Delphi Method,” 10.
member institutions within Christian ministry departments, a homogeneous panel was utilized.
The Delphi Method for This Research
Until this project, little research had yet to be conducted regarding the best practices of spiritual formation in online undergraduate context among CCCU schools.
When this study began, there was no consensus among a panel of experts regarding best practices of online spiritual formation among undergraduate programs. Nor was there explicit research regarding online spiritual formation among CCCU member institutions.
Since the purpose of this study was to identify the exemplars and best practices of
spiritual formation in online undergraduate programs among CCCU member institutions, the Delphi method was implemented to develop such a consensus. The aim of using a Delphi approach in this research project was to work toward a consensus of best practices in online undergraduate spiritual formation. By doing so, this research project provided valuable and identifiable best practices in online spiritual formation for administrators and faculty offering online programs in Christian higher education.
The research instrument and methodology proposed for this study was built on similar work by John Cartwright in his Ed.D. thesis, where he sought to develop
consensus regarding best practices for online theological ministry preparation. Cartwright utilized the Delphi method as his primary methodology for developing such a consensus.
Similar to Cartwright’s work and in line with a Delphi study, the first steps toward verifying and establishing best practices in online undergraduate spiritual formation was to establish a panel of experts.
Delphi panelists. After an initial study of the websites and university catalogs of the 144 CCCU universities, 31 universities were identified as offering online courses or
programs that incorporated intentional spiritual formation elements.28 An initial invitation for participation was sent to 88 faculty and administrators teaching within the Christian Ministry departments at those 31 universities.29 After an initial recruitment phase with emails, phone calls, networking, and follow up, 46 faculty and administrators responded with 26 agreeing to participate. A participant survey was sent to the 26 faculty and administrators who agreed to participate in order to further verify that participants met the research criteria. This initial participant survey verified that professors (1) were faculty or administrators teaching in or overseeing Christian ministry departments, (2) taught among CCCU member institutions, (3) taught at the undergraduate level, (4) had at least five years of experience, (5) had experience teaching or overseeing online courses, (6) had experience teaching or overseeing online courses with intentional spiritual formation elements. Additionally, the initial participant survey30 verified that participants were like minded regarding common characteristics of evangelicalism.31 Prospective panelists were given a week to respond. This initial demographic survey yielded 19 panelists who met the research criteria for this project. Despite multiple attempts to contact, 3
participants did not continue in this research project beyond the initial participant survey.
28Approval for this research project was granted by The Southern Baptist Theological Seminary Ethics Committee prior to any research being conducted.
29See appendix 1 for the initial invitation.
30See appendix 2 for the initial participation survey.
31Robert W. Pazmiño, Foundational Issues in Christian Education: An Introduction in Evangelical Perspective, 3rd ed. (Grand Rapids: Baker, 2008), 57. Gangel references Pazmiño’s summary of evangelicalism as influential in establishing a “full-blown theology of Christian education.” Edward Hayes, “Theological Foundations For Adult Education,” in The Christian Educator’s Handbook on Adult Education, ed. Kenneth O. Gangel and James Wilhoit (Grand Rapids: Baker, 1993), loc. 493, Kindle.
Similar to Pazmiño, John Cartwright articulates four characteristics of evangelical Christianity summarized from D. W. Bebbington’s Evangelicalism in Modern Britain: A History from the 1730s to the 1980s.
Cartwright’s four characteristics include (1) the centrality and authority of the Bible, (2) Jesus’ death as essential to salvation, (3) repentance of sins and faith in Jesus as essential to transformation, and (4) transformation resulting in obedience to God and love of others. John Beck Cartwright, “Best Practices for Online Theological Ministry Preparation: A Delphi Method Study” (Ed.D. thesis, The Southern Baptist Theological Seminary, 2014), 55.
Sixteen participants began this project and by the culmination of the project, 15 participants completed all phases of the research serving as the Delphi panel for this project.
Round 1. After panelists were secured, an initial first round survey of open- ended questions was developed. Whereas Cartwright based his surveys on the learning outcomes established by ATS for Master of Divinity programs, the round 1 survey in this research project was developed on the definition of spiritual formation and the 16
definitional elements established by the CCCU in 2010.32 Utilizing the indices of spiritual formation developed by the CCCU, Round 1 open-ended questions to the panel of experts asked, “How, specifically, do you . . . in your online course?” For example, based on definitional element number 4, “How, specifically, do you foster an awareness of the human condition in your online students?” Or based on definitional element number 9,
“How, specifically, do you encourage the communal and relational nature of spiritual formation in your online students?”33
Qualtrics survey software was used to create the first round survey. Once the round 1 survey was developed it was then pilot tested. Four panelists meeting the research criteria for this project were asked to pilot test the first round survey evaluating it for clarity and functionality. Pilot test participants were given two weeks to provide feedback. The pilot test served as a means to identifying any problems in the survey
32Council for Christian Colleges and Universities, “CCCU Report on Spiritual Formation,”
2011, ix, accessed July 29, 2015, https://www.cccu.org/~/media/filefolder/CCCU-SpiritualFormation_
Booklet.pdf. Hsu and Sandford note, “It is both an acceptable and a common modification of the Delphi process format to use a structured questionnaire in Round 1 that is based upon an extensive review of the literature . . . if basic information concerning the target issue is available and usable.” Hsu and Sandford,
“The Delphi Technique,” 2.
33These questions were adapted from Cartwright, “Best Practices” while utilizing the sixteen definitional elements of spiritual formation developed by the CCCU. Table 1 in chap. 4 includes the full list of adapted questions.
instrument prior to implementation with the entire panel. Three panelists agreed to participate and provided helpful feedback.34
The round 1 survey was sent via a link provided in an email to each panelist.
Anonymity and confidentiality were kept by sending individual emails to each of the 16 participants. Participants were given instructions both in the email and in the initial introduction to the survey. Panelists were given three weeks to complete the round 1 survey. After participants completed the round 1 survey, all answers were collected and collated in an aggregate document. Each question was listed and followed by all 16 answers provided by the Delphi panel. Because there were 16 questions on the round 1 survey and 16 participants, the aggregate document provided all 256 answers to the round 1 survey. Participants were then provided a copy of this aggregate document and a document with their specific answers to the round 1 survey. All identifying information was carefully removed to maintain anonymity and confidentiality. Participants were asked to review their own answers in light of the entire panel and provide any revisions or additions.
Participants were given one week to revise their answers. This revision phase allowed panelists the opportunity to read their answers in light of other panelists and make any revisions or additions as desired. Panelists were not obligated to revise but were provided that opportunity in order to clarify answers and build toward consensus.
Once the round 1 revision phase was complete, the qualitative data provided by the Delphi panel was then imported into NVivo software. The aggregate document with all 256 answers from the Delphi panel was divided by question and imported into NVivo creating a separate document and folder for each question. Because each question was developed from one of the 16 indices of spiritual formation, the NVivo folders served as a means of sorting all spiritual formation practices associated with each indices. Afterward, each question and corresponding set of answers were statistically analyzed and coded
34See appendix 3 for the round 1 survey that was developed and sent to the 16 Delphi panelists.
according to themes and frequency counts that emerged. These themes, or nodes, were then used to develop the round 2 survey. The results of the round 1 survey yielded 78 thematic practices grouped according to the 16 indices of spiritual formation developed by the CCCU. Tables 2 through 17 in chapter 4 list the 78 practices that emerged and the percentage of respondents that cited those practices. Those 78 thematic statements were then used to create the round 2 survey. The round 2 and 3 surveys served as the
quantitative portion of this mixed method research.
Round 2. After answers provided in the round 1 survey were analyzed and the 78 emerging themes were documented, the next step in this Delphi Study employed a four-point Likert type scale asking panelists to rate the importance of each of the 78 thematic statements to their online classrooms as it related to successfully meeting and/or achieving the indices with which it was associated (i.e., how important are the following items when it comes to ensuring your online students understand spiritual formation as God-initiated, Christ-centered, and Holy Spirit-led?).35 Qualtrics survey software was used to create the round 2 survey. In the round 2 survey, panelists were asked to rate each item on its level of importance (1 – Not at all important, 2 – Somewhat important, 3 – Very important, 4 – Extremely important). Appendix 4 displays the round 2 survey provided to participants.
The round 2 Likert-type scale was developed from the emerging themes in round 1 and as with round 1 it was pilot tested prior to use with the Delphi panel. The same four panelists asked to pilot test round 1 were asked to pilot test the round 2 survey.
All four panelists provided feedback related to clarity and functionality. Pilot test
35Hsu and Sandford note that the literature base “strongly favors” a four-point Likert-scale as the preferred second round method of rating answers provided from the first round results. Hsu and Sandford, “The Delphi Technique,” 3-4. Cartwright and Green serve as two examples of using a four-point Likert scale as the round 2 instrument. Cartwright, “Best Practices,” 49; Paul Green, “The Content of a College-Level Outdoor Leadership Course,” University of Oregon, 1982, 24, accessed June 16, 2016, http://eric.ed.gov/?id=ED276546.