S. The nature of sources
4.4 Limitations of the data collection methods
Two major limitations of this study were that it neither accounted for audience contribution nor all online reporting formats used to construct a journalistic framework for reporting on LGBTIQ issues online. The research was formulated in such a way that these were not negligible limitations. However, future studies may involve online audiences to enhance understanding of how frames impact their perceptions and attitudes regarding a specific theme or a similar topic to fully realise the online journalistic frame-building objectives of this study. Moreover, case studies as a data collection approach do not allow for the generalisation of results to the wider population and the researcher’s own subjective feeling may influence outcomes pertaining to a specific case. The process of data collection in case studies is also difficult to replicate and may be time consuming (Reis, 2005). Hannah Schebesta (2018) also argues that both content and thematic analyses are unable to serve as a sole basis for claims about media effects, and therefore researchers who use different tools of measurement may arrive at different conclusions. In the current research, the effects of online reports on media audiences were not explored, nor did this research investigate how online media reside in functionalism, and therefore these limitations were tangential to the research. The research addresses the issue of measurement transferability in the Ethics section of this chapter. Being aware of these limitations and addressing
134 them as they arose (where relevant) was the best strategy to limit their effects on the reliability, validity and trustworthiness of this research project.
A content and thematic data collection and analysis approach can directly investigate online communication − that is, it may determine whether it complies with good reporting practices, creates activism, or fails to create salience, via texts. This is an unobtrusive means of analysing interactions. A limitation of this data collection method, however, is that natural language processing is still under- investigated and can be highly intensive and error prone. The research practiced intervention strategies to prevent this limitation from influencing the results by being clear on what codes were being used to collate data within specific categories or frames.
An issue that needed to be considered was that the size of the population was too voluminous to ensure an accurate mechanistic procedure for the selection, coding and tabulation of the data, and thus a very specific subset of parameters had to be developed to funnel the large number of online reports to an accessible population that would elicit significant data to address the objectives of the research. Another limitation was that the thesis was conducted cross-sectionally, where a longitudinal investigation might have been better to ensure the generalisability of the findings to a wider population. Such an investigation would also have replaced trends by more confirmable patterns in terms of how online reporters deal with LGBTIQ issues. This issue was dealt with by specifying the limitations of the study to contextualise how the findings might be applied to the research problem in order to answer those research questions. The repository nature of the Internet in storing data addressed this limitation to an extent, as the research was able to include all online reports that had been produced (written) at different time junctures at one point and the study was therefore not disadvantaged by engaging in a lengthy inquiry.
The methodology of the research provides important insights into how the research was conducted in the way that it gathers data and subsequently processed that data. These elements are discussed in the conceptual design section where the critical relativist paradigm anchors the relevant design aspects, such as mixed methodology, explorative, inductive reasoning, and cross-sectional time dimension of the research. These aspects are described along with the aspects surrounding the social artefact, namely online reports on ‘queercide’ of four case studies. An in-depth description of the pilot study to address erroneous elements of the data collection instrument (that is, the qualitative coding sheet) as well as the ethical philosophy regarding the methods used in the research conclude the chapter, which is a fundamental part of collecting the actual data. A coding sheet is used to collect relevant data because the online reports do not necessarily present a discourse more indicative of social media, therefore this is a more reliable choice of instrument to collect the necessary data to address the research questions.
135 4.5 Pilot study
To test the clarity and consistency of the qualitative coding sheet that was used as a tool to address all the issues related to the coding categories that needed to be resolved, a pilot study was performed by myself and one other coder who had been trained to apply the coding sheet to the same sample group (Du Plooy-Cilliers et al., 2014). We performed these coding strategies separately and then discussed our respective findings to highlight discrepancies using a moderation form. Ultimately the qualitative coding sheet was adjusted before commencing with thematic coding by using the coding sheet for all 16 selected online reports. After a review of the frames included in the coding sheet and a re- consideration of the Theoretical Framework and Literature Review chapters, two more frames were included. These were whether or not the online reports had a ‘share’ feature as a social plugin (Almgren
& Olsson, 2016; Entman, 1991). This was included to establish a juncture between Newsworthiness and the Framing and Representation Theory.
The following (Table 3) is a copy of the moderation form that was used as part of the pilot study to improve the qualitative coding sheet.
Table 3: Copy of moderation sheet used in the pilot study INTER-CODER MODERATION CONTROL SHEET MODERATION TYPE: Qualitative coding in Pilot Study DATE OF MODERATION
MEETING: 24 August 2019
DATE OF ADDITIONAL
TRAINING: 31 August 2019
PRINCIPAL CODER
NAME: Marchant van der Schyff
SECONDARY CODER
NAME: Kayley Webster
CONTEXTUAL FACTORS TO CONSIDER
Yellow highlighting indicates the discrepancies experienced from inter- coder discussion.
Each Coder coded four articles (one from each case study as identified by the qualitative sample selection process) separately after initial coder training. Thereafter, the principal and secondary coders came together to compare their findings and discuss discrepancies to adjust the coding sheet to be used on the 16 selected articles (four from each case study).
136 The four reports used in the pilot study were not included in the final qualitative study.
Standardisation Meeting 7 September 2019
STUDENT DETAILS
(NAME, SURNAME AND STUDENT NO.)
PRINCIPAL CODER’S REMARKS
SECONDARY CODER’S REMARKS
AGREED CODING
ARTICLE A
K K2 x10
K K2 x10 K2 x10
L Only L1 L Mostly L1
One L2
Mostly L1 One L2 M M8 + M32 +
M29 M M8 + M32 +
M29
Now M8, M32 and M29
N N1 N N1 N1
O O8 O
O8 08
Oa Oa3 Oa Oa3 Oa3
P P1 P P1 P1
Q Q2 Q Q2 Q2
R R3 x2 R R3 x2 R3 x3
S S2 x2
S3 x4 S S2 x2
S3 x4
S2 x2 S3 x4 T At this stage
none identified T At this stage none identified
At this stage none identified
ARTICLE B K
K3 x 1 K5 x 1
K7 x 3 K
K3 x 1 K5 x 1 K7 x 3
K3 x 1 K5 x 1
K7 x 3 (include
‘black’ as other reference)
L L1 x4 L
L1 x4 L1 x4
137 M M7
M32 M
M7
M26 (not ‘sex’
not referring to victim’)
M32
M7 M32
N N2 N N2 N2
O O7 O O7 O7
Oa Oa1 Oa Oa1 Oa1
P
P1 (‘murder’
incorrect, opinion piece)
P P5 P5
Q Q2 Q Q2 Q2
R R4 R R4 R4
S S1 S S1 S1
T T1 T No T codes used T1
ARTICLE C
K
K6 x2
K5 x2 K
K6 x2 K5 x2
K6, but inaccurate as
‘victims’ here refers to Joey and Anisha, not lesbians
L L2 x1 L L2 x1 L2 x1
M M29 M M29 M29
N N2 N3
N N2 N3
N2 N3
O O7 O
O7 O7
Oa Oa1 Oa Oa1 Oa1
138 P P6 x3
P P6 x3 P5
P6 x3, not P5 because ‘murder’
used as legal term
Q Q2 Q Q2 Q2
R R3 x3 R R3 x3 R3 x3
S S2 S S2 S2
T T4 T No T codes used T4
ARTICLE D
K K3 x3
K K3 x3 K3 x3
L L1 x6 L
L1 x6 L1 x6
M M27
M32 M M32
M27 and M32 because ‘Banyana Banyana’ soccer player is high profile
N N1 N
N1 N1
O
O1 O
O1 O1
Oa Oa1
Oa Oa1 Oa1
P No P coded
P P1 P1
Q Q2 Q Q2 Q2
R R2 R R2 R2
S S1 S S1 S1
T T4 T T4 T4
Description of discrepancies found (this section explicitly describes the variances per question)
1) Too many news values and overlapping make assigning codes problematic and eventually discrepant.
2) “other” coding might cause confusion 3) “sex” as news value misidentified
4) “murder” misidentified – through discussion no fix required 5) The use of “victims” creates confusion
6) “murder” misidentified – through discussion no fix required
139 7) news value of “high- profile person” not identified – through discussion no fix required
8) no ‘P code’ identified by primary coder.
Action taken to fix
(This section explicitly describes how the above was addressed)
1) Eliminate non-identified news values in pilot study; combine Galtung
& Ruge (1965), Jewkes (2004), Collins & James (2017) to simplify news values and create mutually exclusive codes
2) Standardise approach: all pronouns referring to victims
3) Clarify “sex” news value and add clarification from Jewkes (2004) 5) Move “victim” to description of victim theme for L (from K) 8) Coding system needs to be managed carefully by test-retesting
Final notes: Coding sheet for quantitative coding adjusted as a result of emergent codes from pilot study as well as inter-coder moderation.
Problematic: Because of the population frame – from date of killing/incident to first court appearance of suspect(s) – the word
‘murder’ as a legal term is accurate (“to test motivation for killing”);
however, outside of connection to case it would be inaccurate and has to be referred to as “killing”, “incident” or “death”.
Problematic: (P) Assumptions, both from writer as well as quoted source because writer still chose to include quote, leaves impression with reader – report framed.
Inclusions: Including the Frame ‘share’ feature (assigned as Oa) will change to P, which has a knock-on effect on the letter assignment of each subsequent frame (for example: “P. Accuracy of online reports”, now becomes “Q. Accuracy of online reports”)
PRINCIPAL CODER PRINT NAME & SIGN:
SECONDARY CODER PRINT NAME & SIGN:
140 The pilot study, which addressed changes and their implications for the study as well as significant details about the collection of data, revealed eight discrepancies from 44 codes. The strategies to resolve these discrepancies through redesigning the coding sheet was summarised above. A significant point that was revealed by doing the pilot study was that there was a high inter-coder reliability from the coding sheet at 82% or (100/44*36(44-8)). It was also found that the frames “Infotainment” (I) and
“Informational” (J) were superfluous because the quantitative coding sheet eliminated the association of ‘queercide’ with entertainment. Therefore, those two frames were excluded from the final qualitative coding sheet. The resulting qualitative coding sheet that was used for data collection from the 16 online reports was the following:
Table 4: Qualitative coding sheet template after the pilot study
CODING SCHEDULE TO ANALYSE FRAMES USED TO REPRESENT
‘QUEERCIDE’ IN ONLINE REPORTING
ONLINE REPORT NAME AND DATE:
FRAME DESCRIPTION OF FRAME
Qualitative: Narratives used in online reports referring to efficacy
K. Moniker used