CHAPTER 4 METHOD DEVELOPMENT AND RATIONALE
4.4 RESEARCH REPORTS SUITABLE FOR CRITICAL ANALYSIS
Methods described in this section address Research question 1: What is the nature of research published on student difficulties with acid-base chemistry? Protocols for a systematic review guided choice of methods, which should show replicable and effective processes for the stages of searching, screening, finding the scope of research, and quality appraisal of data (Torgerson, 2003, pp27-28 & 34-39). In order to analyse research into student conceptual difficulties in acid-base chemistry, it was necessary to first define three sub-questions, which are:
1a) Which reports give suitable research data on student conceptual difficulties in acid-base chemistry?
1b) What is the scope of this research?
1c) What is the overall quality of this research?
These sub-questions were each addressed in sequence, as described in the following sections.
The protocol corresponds to the left hand portion of Figure 4.1.
4.4.1 Developing a search strategy and inclusion/exclusion criteria
Research question 1a was addressed through a variety of search techniques: electronic and hand searches and personal contact, as recommended by Bennett et al. (2005b) followed by screening using criteria developed by the researcher. Searching entailed firstly obtaining papers cited in published reviews (Kind, 2004, and Garnett et al., 1995) then pursuing appropriate references from these cited papers. Next, an extensive search of academic databases (Academic Search Complete, ERIC and PsycINFO) was carried out. The literature review (Section 2.2) showed that a variety of terms can be used to describe student difficulties. Consequently, a variety of keywords and phrases were used in the search, which were: chemistry, acid/s, base/s, Brønsted, Arrhenius, student understanding, student conception/s, alternative conception/s and misconception/s. Then the same keywords were also used with the Google Scholar search engine (www.scholar.google.com). A third aspect of the search entailed systematically searching (by hand or electronically), as far back as 1978, the indexes and tables of content from science education journals available through the University of KwaZulu-Natal library. These included International Journal of Science Education, Journal of Research in Science Teaching, Science Education, Journal of Chemical Education, Chemical Education Research and Practice and, finally, Research in Science Education. Additionally, I was fortunate to be able to obtain some pre-publication copies of research reports through personal contacts. The abstracts of all suitable papers were scrutinized and if possibly suitable, hard copies were obtained. Finally, where authors of these papers had cited work that might have been suitable, these too were
obtained. I judged the search had reached saturation point when further iterations of the process showed the same reports. This search took place between May 2006 and January 2008.
As discussed above, inclusion or exclusion of a study for a review should be based on clear criteria (Eybe & Schmidt, 2001; Bennett et al., 2005b). The review publications described earlier (Section 4.2) had not suggested any suitable criteria, so it was necessary to derive my own. These were driven by the research aim and first research question, and guided by the theoretical framework and advice regarding systematic reviews (Torgerson, 2003; Bennett et al., 2005b). Chosen criteria are given below in Table 4.1 (in flip-out form for readers’
convenience) followed by the rationale behind the choice of each criterion.
Table 4.1 Inclusion and exclusion criteria for research publications
Aspect Inclusion if research includes: Exclusion if Research is limited to:
1 Chemistry concepts
Acid-base reactions Underlying principles, e.g. equilibrium, formulae
Redox reactions of acids or bases Applications of acid-base chemistry, e.g.
environmental or physiological 2 Acid-base
models
Operational model (macroscopic properties)
Arrhenius & Brønsted.
Other models, e.g. Lavoisier (historical), Lewis (not generally high school)
3 Type of knowledge
Conceptual knowledge Isolated facts, e.g. indicators colour
4 Type of students
Any of:
Elementary to post-graduate students, and teachers
Laypersons, other professionals
5 Research aims or questions
Probing for, or identification of student conceptions in an educational setting, pre- or post- instruction
Purely quantitative studies on prevalence or achievement.
Instructional programmes 6 Type of
research data
Student quotes or author knowledge claims about nature of conceptions, not previously published by the authors.
Data of similar nature, from publications which are not available
Textbook quotes
7 Publication date
January 1978 to December 2007 Before January 1978 and after December 2007
8 Language of Publication
English Other languages
9 Type of publication
Journals, available through academic libraries in South Africa or
electronically
Conference Papers published on www.
Theses, Conference proceedings not freely available on www or through South African academic libraries
Criterion 1
A narrowly focused research question is recommended by Torgerson (2003) and Bennett et al.
(2005b). Accordingly, this study was limited to student difficulties in acid-base chemistry.
Although these could be caused by problems with underlying concepts (Furió-Más, et al., 2007), such difficulties with more fundamental ideas such as distinguishing a chemical reaction from simple mixing, understanding the nature of chemical equilibrium or writing chemical formulae were not investigated. Redox reactions of acids or bases require different models for explanations to those described in the chemistry context (Section 3.3), so they were excluded; as were environmental and physiological applications of acid-base chemistry. Despite the importance of such problems as acid-rain, it was judged too broad to also target environmental or physiological as well as chemical concepts.
Criterion 2
High school curricula seldom embrace the Lewis model of acid-base chemistry (Oversby, 2000a). In order to focus on high-school chemistry, where I had the most experience, only conceptions of the phenomenological or macroscopic aspects of acid-base chemistry, with the Arrhenius and Brønsted models used to explain the phenomena were included. Students entering tertiary education could be presumed to have studied and mastered these conceptions at high school (see Section 3.3).
Criterion 3
The search for data on student difficulties with acid-base chemistry also focused on conceptual understanding as described in Section 3.2.2 on meaningful learning. For example, isolated items of arbitrary knowledge such as the colour change of particular indicators would not be included as these would need to be learned by rote, whereas understanding how indicators work and the choice of indicators for titrations of weak acids or bases could be included.
Criterion 4
Previous studies into student conceptions were included, provided they were not simply a survey of laypersons or other professionals such as nurses or engineers, who might not have received formal instruction in the relevant topics. There was no restriction on the age and educational level where conceptions were researched; indeed it would be advantageous to include a wide range of ages and contexts to ensure enough representative evidence for a conception (Grayson et al., 2001; Liu, 2001). Furthermore, as discussed under the ‘student and course context’ (see Section 3.5), student difficulties are widespread and tenacious, even into tertiary education, so that conceptions developed earlier in a students’ career may still have an impact on how students filter new educational experiences later on. Consequently, this study could cover elementary school through to tertiary level and post-graduates, also including
professional teachers. I include all these individuals under the banner of students when discussing student difficulties, as even teacher understanding is most likely to have developed during their student years.
Criterion 5
Any project that included an investigation into student difficulties, whether this was a primary aim or a necessary part of evaluating the effectiveness of an intervention has been included, provided there was suitable data available (see criterion 6). Pre-instruction data would need to be interpreted judiciously, as conceptual difficulties with chemistry models are not principally due to intuitive pre-conceptions (Taber, 2001a).
Criterion 6
With a view to gaining greater insight into the nature of particular student conceptions, rather than prevalence, I anticipated collecting any suitable qualitative data from the published papers in the form of student quotes or author knowledge claims, or perhaps in the form of distractors for multiple-choice items. Tan et al. (2002) considered alternative conceptions significant when there was a 10% incidence in their sample group, but this was not used as a criterion for this study, because research cohorts might be very small or an investigation only exploratory through open ended questions. However, data on the prevalence of a conception or student achievement data that was purely quantitative was deemed irrelevant in this review if it shed no further light on student understanding. Some authors cited and quoted descriptions of student difficulties from other publications – including internal reports from their own research group.
These secondary sources have been included only when the original reports could not be obtained. We also excluded research focusing on poor presentation of this topic in textbooks (e.g. Drechsler & Schmidt, 2005a). While textbooks are undoubtedly one source of student misconceptions, such a content analysis represents a different research project. Similarly, research into the effect of different teaching strategies also fell outside this project. As with textbooks, this would constitute a separate project in itself.
Criterion 7
Publications over a thirty-year period were included. Driver and Easley’s (1978) seminal work on student conceptions marks the start of this review and I looked for no further publications after December 2007. I included reports obtained through personal contact, at that stage ‘in press’, so their publication dates might be later than 2007.
Criterion 8
Only research published in English was included as I could not have done justice to work published in another language. In such cases, secondary reports on the work were accepted if they were available.
Criterion 9
Torgerson (2003) advocates including worthwhile smaller studies not published in the main journals, but still in the public domain. For logistical reasons, sources were limited to those available through academic libraries in South Africa or in the public domain on the World Wide Web. As a result, some research in dissertations, internal reports and certain conference proceedings was only available through secondary sources, or not at all.
As each research report became available, those that met the acceptance criteria for analysis were allocated numerical codes. These enabled me to later ‘tag’ each piece of data back to its source and to link all reports from one research group – important because they could involve overlapping data. The codes followed no particular order, simply being allocated in sequence as the reports were obtained. Information was initially recorded by hand on a separate printed form for each research report, and then summarized on a Microsoft Excel spreadsheet. This completed the selecting phase of answering Research question 1a, the results of which are given in Section 5.2 of the next chapter. Bibliographic information (according to column 1 in Table 4.2 below) for all the suitable research reports is given in Appendix 1. These reports were then used to categorise the scope and quality of the existing research in the next phase.
4.4.2 Categorising data on the research reports.
Qualitative research data should be interpreted within context (Lincoln & Guba, 1985, p 42) and, being guided by Torgerson (2003, pp 45-47) and Bennett et al. (2005), I extracted contextual information, as shown below in Table 4.2, from each published report, if it was published. A matrix, as suggested by Çalýk et al. (2005), was used to summarize the data. The data extraction and coding ran concurrently with the search process (see above).
Table 4.2 Contextual information extracted from suitable reports
Bibliographic information Context of study Research methods Report code
Search Source,
e.g. ERIC or Google Scholar Author(s)
Year Published Title
Full Citation
Country Date of the study Research aims Educational setting
Pre-instruction or post-instruction?
Participant details:
e.g. age/ educational level, gender, ethnicity, socio-economic level
Acid-base topics
Cohort size
Data collection instrument(s) Probes or interview questions given?
In what form is propositional knowledge given?
During the categorising phase, by looking at the contexts covered by the body of research, I first sought an answer to Research sub-question 1b: What is the scope of this research? The results
of this analysis are given in Section 5.2 of the next chapter. The next category of data concerned information on research methods in order to answer Research question 1c: What is the overall quality of this research? Codes indicating aspects of research methods such as the data collection instruments or nature of propositional knowledge given were used as suggested by Ünal et al. (2006). Sometimes, not all the desirable information was published. The research reports were then critiqued using guidelines published by Eybe & Schmidt (2001) to indicate the overall quality of the research. This is presented in Section 5.4 of the next chapter.
Assessment of the overall quality of the body of research informed the methods adopted for answering Research questions 2, 3 and 4 as described in the next section. Categories of acid- base topics informed the initial categories of student difficulties and propositional knowledge (Research sub-questions 2a, 3a and 4a, and 2c, 3c and 4c, see Sections 4.4.2 and 4.5.2 respectively) while the data on research methods were used in evaluating the stability of difficulty descriptions as described in Section 4.5.5.