4.11 PHASE 3 : EVALUATION OF INTERVENTION (Outcome and Process Evaluation Methodology)
4.11.1 Outcome Evaluation
145
4.11 PHASE 3 : EVALUATION OF INTERVENTION (Outcome and Process
146
To inform the evaluation phase of this study, it was deemed necessary to locate outcome indicators at the different levels of influence i.e. the community school level, the interpersonal level and the intrapersonal level to understand their role in promoting a multi-systemic, whole school approach to school connectedness.
Outcome evaluation
In relation to evaluating outcomes of objective 1 and 2 i.e. to
Enhance adolescent learners mental health well-being and connectivity to the school
To strengthen adolescent learners academic motivation
A quasi-experimental approach using a before and after matched control design was utilised to determine whether the intervention had the intended outcome on the study participants.
Study site
The intervention and control school site are already discussed in the Methodology Chapter, section 4.8.1
4.11.1.1 Data collection objectives and measures
The outcome measures used to assess the first objective i.e. enhance adolescent learners mental health well-being and connectivity to the school at the community school level was the Psychological Sense of School Membership (PSSM) Scale.
The outcome measures used to assess the second objective i.e. to strengthen adolescent learners academic motivation at the intrapersonal level was the
147
Future Orientation Scale which comprised two subscales i.e. the Reason for Achievement Scale and Value of Academic Success Scale.
4.11.1.2 Administration procedures
After ethical clearance was approved at both the intervention and control school, the school principal and Life Orientation teachers assisted the researcher in identifying time in the school time table to administer the surveys to Grade 10 learners. The school management team thus arranged for all grade 10 learners to be administered the survey at a designated time and venue. The surveys were administered and collected from the entire grade at both the intervention and control school during a lunch break period
4.11.1.3 Pre-post administration and Sampling
The sample thus compromised all grade 10 learners from both the intervention and control school. The sample thus included 137, pre-intervention surveys using the PSSM and FOS scales were administered and collected, to learners at the intervention school in March 2011. 123 pre-intervention surveys using the PSSM and FOS scales were administered and collected at the control school in March 2011. 137 post intervention surveys using the PSSM and FOS scales were administered and collected at the intervention school in November 2011 and 123 post-intervention surveys using the PSSM and FOS scales were administered and collected at the control school in November 2011.
4.11.1.4 Psychological Sense of School Membership (PSSM)
The Psychological Sense of School Membership (PSSM) developed by Goodenow (1993) is an instrument used by school mental health researchers and practitioners.
This 18-item instrument was developed to investigate school membership of learners
148
in secondary schools in the US. This scale was utilized as a measure of school connectedness and belonging in this study given the central role played by learners’
sense of belonging to school connectedness as discussed in the introductory chapter.
The items were phrased in terms of individual frame of reference and probed for a global factor, namely, the subjective sense of belonging in school. The items on the questionnaire are scored on in a 5-point Likert Scale ranging from 1=not at all true through to 5 = completely true. A study by Hagborg (1994) focusing on US middle and upper secondary school students, indicated that the items of the PSSM scale can be classified into three different factors-belonging, rejection, and acceptance. The first factor (belonging), included 13 of the 18 items, presented a general aspect of school membership. The second factor (rejection), included 3 items, and dealt with students’
feelings of personal acceptance among their classmates. The third factor (acceptance), which included only 2 items, measured student acceptance of and pride in their school.
4.11.1.5 Reliability
The Psychological Sense of School membership (PSSM) is a well used measure by various researchers globally. Research has indicated at least 41 studies have used the PSSM scale. Fifteen of these investigations used an abbreviated version specific to each study and the other 27 studies employed the full 18-item scale (You, et al., 2011). The Cronbach’s alphas were reported in previous studies had ranged from 0.78 and 0.95 across samples of elementary and secondary school students from diverse backgrounds including African Americans, Chinese, Hispanic Americans, Israeli’s, Native Americans, Somali’s, refugees in the USA, and USA European Americans (Goodenow, 1993) Hagborg (1994) reports high test-retest reliability (.78) across four
149
weeks. Other researchers such as Shochet et al., (2006) found that PSSM scores were relatively stable across time in a clinical intervention study, where a 12month test- retest correlations of 0.56 and 0.60 was found for boys and girls respectively. In recent South African studies, Govender et al., (2013) and Basterfield, Reardon, and Govender (2014) found Crobach alphas of 0.84 and 0.76 respectively among school going adolescents in Durban, KwaZulu-Natal.
4.11.1.6 Academic Motivation
In evaluating learners sense of Academic Motivation, the Future Orientation Scale (FOS) combined two sub-scales assessing students educational and academic values.
The first sub-scale called Reason for Achievement Scale measures academic motivation (Kuperminc, Darnell, Jurkovic, 2004). It includes 14 items assessing the degree to which youth value school and their level of academic commitment. The second sub-scale, which includes three items, was adapted from the Value of Academic success scale developed by Fuligni (1997). These three items measured the perceived importance of getting good grades, finishing high school and going to university. When testing for the impact of the intervention on future orientation, these two sub-scales were combined into a seventeen item Future Orientation Scale (FOS).
See Appendix 6b. for the FOS Scale.
4.11.1.7 Reliability
Both sub-scales have been identified as possessing good internal consistency and being equally reliable across the different ethnic backgrounds (Fuligni, 1997);
(Kuperminc, Darnell, Alvarez-Jimenez, 2008). In this study, the Cronbach’s alpha
150
coefficient of the combined FOS scale was .68 indicating acceptable internal reliability.
4.11.1.8 Data Analysis
Statistical data analysis was performed with the aid of the software; Statistical Package for the Social Sciences (SPSS) version 21. Initial data entry was done with the help of a research assistant. Data was entered in this software using numerical codes for all variables except the age of respondents which were entered in their original form. For the demographic variables, gender was coded as 1=male and 2=female; age was entered in its original form. Population group was coded as 1=African; 2=White; 3=Indian; and 4=Coloured. All 18 items from Goodenow’s (1993) PSSM scale were administered. The scale range was from 1 to 5 (1 referring to not at all true and 5 means completely true). Items 3, 6, 9, 12, and 16 were negative items and were reverse coded. The second scale, the Future Orientation Scale which comprised items assessing the degree to which youth value school and their level of academic commitment.
Descriptive statistics including frequencies, percentages, means and standard deviations were performed following which repeated measures analysis of variance (ANOVA) was conducted on the main outcome measures (PSSM and the FOS), to examine the within group and between group differences between the pre-test and post-test outcome measures.
151 4.12 Process Evaluation
For the process evaluation component of this study, qualitative methods were used to monitor the reliability of the programme implementation and to understand the factors mediating the findings of the outcome evaluation. These methods included:
Regular meetings with key role players (Principal, school managers, Life orientation teachers)
Workshop on developing the Schools Liaison Group
Workshop on implementing the lesson plans and monitoring
Review of lesson activity sheets of learners
Development and Implementation of the Peer Mentoring Programme
? individual interviews with key role players (managers, teachers)
Four focus-group discussions with adolescent learners The process evaluation section included the
Data collected via these qualitative methods was analysed thematically, following the method proposed in the Methodology section.