Chapter 5 Methodology Methodology
5.5. Reliability and Validity of the Study 1 Reliability. 1 Reliability
demographic information eliciting marital status of participants. This category was therefore added in the later version of the questionnaire used in this study.
5.4.5. What was learnt from the pilot study?
There are a number of important lessons that emerged from the pilot study.
Apart from the adaptations made to the questionnaire as stated above, the pilot study proved important in terms of the significance of thorough training of the research assistants, teamwork, and short briefing meetings every morning prior to proceeding with data collection. The importance of establishing rapport with the research participants was also learnt from the pilot study. Research assistants noticed that participants were more likely to answer the questionnaire in full when a good rapport was established, when participants understood what the study is all about and when the participants were informed about how important it would be for their ideas to be included in the study, as well as when they were informed from the beginning that close to an hour of their time was required to complete the questionnaire.
5.5. Reliability and Validity of the Study
There are a number of methodologies available to test for the reliability of a questionnaire. The most commonly used method by quantitative researchers is through the process of calculating inter-item consistency (Dunn, 2010). The researcher in this study used Cronbach’s alpha on data obtained from the pilot study to calculate the internal consistency of the items comprising the questionnaire. According to Clark-Carter (2010), Rosnow and Rosenthal (1996), as well as Stead (2001), Cronbach’s alpha is a commonly-used measure of internal- consistency reliability that is used with questionnaires based on a Likert scale. The reliability of the questionnaire for this study was anticipated to be confirmed with a Cronbach’s alpha coefficient greater than 0.7 (Coolican, 2004; de Vaus, 2002), and the closer to 1 the better because high internal consistency would mean that the items were all measuring the same construct or idea (Devlin, 2006). The Cronbach’s alpha (α) coefficient for this study was .89 for the pilot study, and .85 for the main study. The higher values, typically above +.80, according to Dunn (2010), are desirable because they indicate greater consistency.
5.5.2. Validity.
In addition to reliability, the researcher also ensured that the questionnaire employed in this study is valid. Validity refers to whether a measure is truthful or genuine (Jackson, 2012), or in other words, whether the questionnaire measures what it claims to measure. This, according to Stead (2001), refers to the extent to which a research design is scientifically sound or appropriately conducted.
Different types of validity are usually examined in a research study. Three important ones that are relevant for this study are:
(1) Face validity, which is the degree to which the test seems to measure the appropriate concept at face value (Weisberg et al., 1996). This concerns whether the questionnaire appears to measure what it claims to measure at face value. Stead (2001) argues that if the questionnaire lacks face-value, the research participants may question the purpose of completing the questionnaire. The three psychologists who were asked to review the vignettes used in this study, as well as the five people who participated in the pre-testing of the questionnaire, all agreed that the questionnaire appeared to measure what it purported to measure.
(2) Content validity is described by Jackson (2012) as the extent to which a measuring instrument covers a representative sample of the domain of behaviours to be measured. In this study, different questions were used to measure different
aspects of knowledge and attitudes toward mental health and help-seeking behaviours. According to Dunn (2010), a conventional approach to determining the content validity of a questionnaire is to have it reviewed by a panel of expert judges in the field of enquiry. This was carried out by the three psychologists who had also confirmed the face validity of the questionnaire of this study.
(3) Construct validity refers to the degree to which a test measures the theoretical construct or abstract variable it was intended to measure (Stead, 2001).
This means that construct validity is related to the theoretical knowledge of the concept being measured. While a construct is an abstraction of something that cannot be seen, it can be inferred because, according to Babbie (2011), it is based on the logical relationship among variables. Babbie (2011) adds that this can be measured by developing certain theoretical expectations about the way in which a certain variable relates to other variables. For example, in this study better knowledge of mental illness is expected to relate more closely with less stigmatising attitudes towards mental illness and improved help-seeking behaviours. To improve construct validity, given that the questionnaire used in this study measures different aspects of mental health literacy, various items measuring a particular aspect (for example, knowledge of treatment options for mental illness) were arranged to be aligned together, for example, to form knowledge of treatments that are likely to be helpful for mental illness. This procedure is recommended by Muijs (2011).
According to Czaja and Blair (2005), validity also requires that the respondents interpret the question as intended. Validity in this manner was tested during the pre-testing of the questionnaire in this study.
The following guidelines to improve the validity of the questionnaire, provided by Stangor (2004), were taken into consideration during piloting of this study: 1) making sure that questions are understandable and not ambiguous; 2) attempting to get respondents to take the questionnaire seriously by emphasising that the honesty of their responses is important; 3) making items nonreactive; 4) choosing items that seem ‘reasonable’ and which represent a broad range of questions concerning the topic of interest; and 5) using an existing scale in which reliability and validity have already being established. Furthermore, the three psychologists who were asked to check the vignettes of this study were also asked to ensure whether they think the questionnaire meets these three kinds of validity and whether they were all in agreement that it does.