CHAPTER 5 RESULTS SHOWING SCOPE AND QUALITY OF RESEARCH ON
5.4 QUALITY OF THE RESEARCH
5.4.2 Problems with reported research
The problem does not lie in these research projects, but the uncritical way in which other authors have subsequently cited their results, with no reservation that they came from a single cohort, or even single students. For example, Pinarbasi (2007, p 24) writes “Nakhleh and Krajcik (1994) established that ...” with similar statements in Lin and Chiu (2007). Likewise, Dhindsa (2002, p 21) writes about Ross and Munby’s work as “It has been known that students ...” Moreover, another problem arises when instead of taking this work, and building on it to be able to describe a conception more accurately (as did Schmidt) some authors use these findings without further investigation as distractors in their own multiple-choice probes. In particular, Demicioğlu et al. (2004; 2005) use “All acids have bubbles” (Nakhleh & Krajcik, 1994) without reporting further research. Consequently their data adds little to clarify the nature of a conception except that other students also chose these words. Other conceptions have been reported more recently through similarly triangulated quality research among single student cohorts (e.g. Demerouti et al., 2004; Watters & Watters, 2006; Sheppard, 2006; Furió-Más et al., 2007) and it remains to be seen how these will be subsequently cited. It appears that many later authors treat all reported conceptions as “established” needing no further investigation into their nature.
5.4.2.1 Analysis of types of data collection instruments
In all the 41 selected reports some information on data collection methods was reported, which as can be seen in Table 5.1, included a variety of data collection instruments. Furthermore, in their efforts to obtain data from a variety of sources, or to investigate different chemistry contexts, nearly all the authors report some attempt towards the goal of triangulation (Lincoln &
Guba, 1985; McMillan & Schumacher, 1993), even if they did not achieve full triangulation.
Nearly half of the authors (n = 18) used at least two means of data collection. For example Toplis (1998) reported using both interviews and observations while Zoller, 1996 gives results from examinations answers and interviews. Some authors, however, beyond stating they used a paper and pencil instrument, gave no further details of their instruments, so their research could not be given a fair evaluation (e.g. Hand & Treagust, 1988; Ogunniyi & Mikalsen, 2004).
Where authors reported only one type of instrument, all except one (Ye & Wells, 1998) included some sort of open-ended component or two-tiers of answers (MC2, MC3, MCE see Section 4.5.5.1). This diversity implies that, in line with the principles of triangulation, this analysis could fruitfully combine different investigations to give different perspectives on the same student difficulty; a necessary condition for an accurate description of the difficulty, in accordance with criteria given in Table 4.4 (in flip-out format on page 72) in order for the descriptions to be classified on the third level or above. For example, Botton (1991) (concept mapping), Nakhleh and Krajcik (1994) (interviews) and Demircioğlu et al. (2005) (interviews followed by paper and pencil items) would possibly combine well to show conceptions of the role of acid-base indicators.
Closer analysis of the various data collection instruments shows that interviews (I or Ig) were used to collect data in 21 (51%) studies. Pencil and paper instruments were the most popular choice as reported in 28 (68%) studies; six of which give no further details (pp). Eight (20%) reports included research with open-ended items (OE) and a further nine (22%) of the investigations included modified multiple-choice items; either with free explanations (MCE), or two-tier multiple-choice items requiring both an answer and explanation (MC2) or in one instance three-tier, which also asked for students’ degree of confidence in their answers (MC3).
Multiple-choice items in a conventional format of a stem with one answer and several distractors were used in ten studies (24%). Except for the one noted earlier (Ye & Wells, 1998), it is heartening that where these extremely focused instruments had been used, in all cases they had been coupled with other less focused probes: either interviews or paper and pencil items.
However data from open-ended responses has not always been published in the report. In summary nearly all authors report investigating student conceptions through open-ended means,
and over two thirds used paper and pencil instruments, where probes in a multiple-choice format were preferred. Only half of the projects entailed interviews, but unlike Sheppard (2006) and Furió-Más et al. (2007) who describe interview tasks and questions, very few authors report details of the protocol adopted in interviews. As a consequence one cannot tell if these were conducted within students’ frame of reference.
Methods by which probes were designed have also not been well documented. Many authors describe the procedure by which probes were designed very briefly with few details. For instance, Linke and Venz (1979) simply report that questions had been checked by university physical science teaching staff. Cros et al. (1986; 1988), Ogunniyi and Mikalsen (2004), and Pinarbasi (2007) also report in such broad terms. Even worse, Demircioğlu et al. (2005) assert that their research probes were developed according to Treagust’s (1988) method, yet they describe them as having a correct choice, a common misconception, and three “reasonable and plausible distracters” (p 43). Furthermore the only example given is a classical multiple-choice item. In their report there is no evidence of the two-tier items that characterize Treagust’s procedure, so their claims about the procedure have little substance. Other authors give no sound reasons for including particular items. In this regard, Bradley and Mosimege (1998) simply based their questions on local textbooks, past examination papers, and teacher experience. These glib claims about research procedures contrast with carefully documented details of validity and reliability checks as reported by Demerouti et al. (2004). As a result of inadequately documented research procedures, research findings need to be used with caution.
In summary, almost all reports show that some form of open-ended instrument was included, although procedures for establishing validity and reliability of items are not well documented.
This leads to doubts about the nature of particular items used in research.
5.4.2.2 Nature of research probes
The nature of research probes was analysed next. All relevant paper and pencil probes were available with 20 (49%) of the reports. For some projects these were available as an appendix (e.g. Bradley & Mosimege, 1998) or as supplemental material on the journal website (Furió- Más et al, 2007). Others gave these in part (e.g. Zoller, 1996), but 10 reports (24%) gave none of the probes at all (e.g. Dhindsa, 2002). Consequently, it is impossible to evaluate these. Some research probes are very simple, for example: “Give a definition of ‘acid’” (Cros et al., 1986, p 313) or more complex, involving over 100 words and many technical chemical and biochemical terms (Watters & Watters, 2006). Problems are evident in Bradley and Mosimege’s (1998)
questionnaire which lacks focus in that some items treat acids macroscopically, while others use the Brønsted model, but do not specify this. Similarly Kousathana et al. (2005) report on a multiple-choice item where the stem asked: “Which of the following species cannot act as an amphiprotic substance?” but the distractors included H2O and the formulae for ions: HCOO-, HCO3
- and HS- (my italics). In this regard, the term substance relates to macroscopic representations (elements, compounds, mixtures) as appropriate in the Arrhenius or operational models. By contrast, the term species relates to the sub-microscopic world of particles such as atoms, molecules and ions, appropriate to Brønsted model (Loeffler, 1989). Furthermore, a number of authors (e.g. Andersson, 1990; Selley, 2000) have reported that students frequently ascribe the macroscopic properties of a substance to individual atoms, molecules or ions. Thus it should be no surprise that student conceptions indicate hybrid models, when even research probes are not clear. Moreover, without clear signposts indicating appropriate models, such questions are unlikely to be within students’ frame of reference as advocated by Johnson & Gott (1996). Accordingly, findings from the reports such as these cannot be taken on face value.
5.4.2.3 Data interpretation and propositional knowledge
The way in which data is interpreted also contributes to the validity of the research. Criteria given in Table 4.4 (in flip-out format on page 72) show two aspects that need be appraised in the author’s interpretation of data; these are the context given in probes and the chemistry context used to interpret responses. Both should be within students’ frame of reference (Johnson & Gott, 1996).
Propositional knowledge statements for the chemistry context has not been given due importance in the research reports. Some authors do not even state which acid-base model was being investigated (e.g. Linke & Venz, 1979; Demircioğlu et al., 2004). By contrast Ouertatani et al. (2007) make the context of the Arrhenius model quite clear in their title. Propositional knowledge was completely omitted in 13 reports (30%). Furthermore all but one of these purported to be investigating the nature or prevalence of student conceptions; yet they gave absolutely no indication of what they considered as scientifically acceptable. As described in Sections 2.4.6 and 4.6.1 the nature of science precludes a single unproblematic fixed body of acceptable knowledge which makes it necessary to report propositions against which student conceptions will be judged. In other reports some propositional knowledge could be inferred from a theoretical framework of general scientific principles (10 reports, 24%, e.g. Erduran, 2003) or discussion of results (9 reports, 22%, e.g. Dreschler & Schmidt, 2005a) or some statements specific to the probes (13 reports, 32%, e.g. Pinarbasi et al., 2007). Only one report
gave a list of individual propositional statements (Nakhleh & Krajcik, 1994) and in this there was evidence of mixed models. To elaborate, these authors claim the list represents “a synopsis of the Brønsted-Lowry model of acids and bases found in most high-school texts” (p 1078).
However they devote a section to macroscopic properties such as taste, indicator colours, and titrations, which are not relevant to the model (see Section 3.3.3). Furthermore, they describe bases as proton acceptors, and describe OH– ions as being a typical base, yet they give NaOH as an example of a base. The problem might have arisen in the textbooks from which the statements were gleaned, rather than the researchers. Nevertheless it highlights the urgent need for textbook revision according to sound propositional knowledge. Another problem occurred with scientifically unacceptable statements being given as propositional knowledge. This problem occurred with Botton (1990) where a ‘model’ concept map indicates that strong or weak acid or bases have fixed and characteristic pH values, rather than these values being variable according to the concentration of the substances in solution. The lack of these two important components of the research (qualitative data as student quotations and propositional knowledge statements against which to evaluate these) causes concern. It can result in some researchers making claims about ‘misconceptions’ with little or no evidence to back their claims (e.g. Hand & Treagust, 1988; Demircioğlu et al., 2004; Ouertatani et al., 2007).