CHAPTER 10: SUMMARY, CONCLUSION(S) & RECOMMENDATIONS 199-208
4.7 Reliability and Validity Differentiated
104
officials among others while the second focuses on the salience of the attributes (Kiousis and McCombs, 2004:38).
The concepts of framing and agenda-setting is being applied in this study because it is expected to aid the researcher to study and analyse the pattern of newspaper editorial presentations so as to make an argument into the present-day Ghanaian media and to suggest a way forward. Overall, both framing and attribute agenda-setting pulls attention to the viewpoints of communicators and their audiences, how they perceive headlines in the news and more specifically, to the special prominence that certain attributes or frames have in the message.
105
defined as “the extent to which results are consistent over time and an accurate representation of the total population under study is referred to as reliability and if the results of a study can be reproduced under a similar methodology, then the research instrument is considered to be reliable” (Joppe, 2000:1; cited in Golafshani, 2003:598). Reliability connotes dependability and consistency. It implies the same result is capable of being repeated or recurs under identical or very similar conditions. The converse is when a measurement process that yields erratic, unstable, or inconsistent results (Neuman, 2006:188). Furthermore, reliability is the
degree of consistency shown by one or more coders in classifying content according to defined values on specific variables. Reliability can be demonstrated by assessing the correlation between judgements of the same sample of relevant items made by different coders („inter-coder reliablity‟) or by one coder on different occasions („intra-coder reliability‟) (Bell, 2001:21).
4.7.2 Inter-coder Reliability
Inter-coder reliability is an extensively used term to determine the extent to which coders independently evaluate a feature of a message or object and arrive at the same conclusion.
(Lombard, Snyder-Duch and Bracken, 2004:2). Furthermore, inter-coder reliability is the
“equivalence reliability in content analysis with multiple content coders that require a high degree of consistency across coders” (Neuman, 2006:326).
High reliability levels can be achieved in three main ways according to Bell (2001:22) who states that high levels of reliability occurs when researchers (i) Set precise and clear definition for variables and values and ensure that all assigned definitions are understood by coders similarly; (ii) Ensure during training, coders are thought to practically apply criteria defined for each variable and value and (iii) Carry out inter-coder consistency measurement after two or more coders have applied the defined criteria as coding examples. Agreeing with and further expanding on Bell‟s postulation on achieving „high levels of reliability‟, Lynch and Peer (2002:46) explained four measures for evaluating consistency and reliability. These are, firstly, to forestall future problems, devote much time prior to the entire content analysis process. This ensures that enough training has been given to coders. Secondly, coders to be recruited should be restricted to a small number. This makes managing the group [coders] easy. Consistent with
106
Neuendorf‟s position, “F1: coding” and “E: training and initial reliability” (Figure 4.0) two coders, though experienced, were recruited and trained to code newspapers in the current study.
This training made coders conversant with the task under consideration. During this process, coding instructions and instruments (see appendix 2 and 3) were revised. Thirdly, time frame allotted to coding should be limited to a shorter period as possible. This quickens the coding process; and avoids coders re-studying coding instructions to enhance inter-coder reliability and finally, workspace should be kept away from daily tasks. Making provision for a spacious room and keeping stationery orderly helps coders to focus and raise concerns where need be.
Lynch and Peer further suggested that though scientifically there exist two ways of measuring inter-coder reliability, the best practice is to combine the two. Lynch and Peer‟s technique involves: (i) a set of same story samples is randomly selected and supplied to coders to code. Extracted coding is then compared for consistency and (ii) employing an expert coder to randomly select 10%-20% out of the total stories and after independently coding, compare coding for those stories. The current study, in line with Neuendorf‟s content analysis flowchart (Figure 4.0) labeled “E: training and initial reliability”, the two trained coders selected 10% of the newspapers and coded them independently to ascertain the efficacy of the coding instruction and instrument prior to the main coding which lasted for two days.
The percent agreement, consistent with section “G-Final Reliability” on Figure 4.0 was used to calculate inter-coder reliability as .98% for advertising placement and .92% for stories.
The reason for this is that numerous methods have been formulated for reliability quantification, however, the „percent agreement‟ and „pi‟ are the two that employs the least mathematical approaches in their calculations (Bell, 2001:22) and are therefore much easier.
4.7.3 Validity
Validity is that quality of research results that leads us to accept them as true, as speaking about the real world of people, phenomena, events, experiences, and actions. A measuring instrument is considered valid if it measures what its user claims it measures (Kripendorff, 2004:313). “Validity concerns the confidence we have that we are measuring what we think we are; the accuracy of our results. Do our results actually reflect what is happening or are they due to something else?” (Meadows, 2003:523). Furthermore, validity
107
determines whether the research truly measures that which it was intended to measure or how truthful the research results are. In other words, does the research instrument allow you to hit „the bull‟s eye‟ of your research object? Researchers generally determine validity by asking a series of questions, and will often look for the answers in the research of others. (Joppe, 2000:1; cited in Golafshani, 2003:599).
Thus, validity advocates truthfulness and explains how well an idea “suits” actual reality (Neuman, 2006:188).
In simple terms, validity addresses the question of how well a social reality being measured through research matches with the constructs researchers use to understand it. Perfect reliability and validity are virtually impossible to achieve. Rather, they are ideals researchers strive for (Neuman, 2006:188).