ACTIVE INGREDIENTS
4.2.2 Empirical Approach .1 Overview
The empirical approach is the most frequently advocated and used in designing health interventions, consistent with the emphasis on evidence-based practice.
The approach is appropriate where evidence is available on the usefulness of
4.2 Approaches for Delineating the Intervention’s Active Ingredients
73
interventions in addressing the health problem. The empirical approach relies on evidence to identify and select interventions. The evidence, derived or synthesized from intervention evaluation research, points to therapies or techniques that have been found effective in addressing specific aspects of the problem or the overall problem. Traditionally, empirical evidence is generated in studies that evaluated an intervention as a whole package. Reports of these studies highlight the results pertaining to the direct effects of the intervention on the health problem. The reports fall short of describing: (1) the intervention itself (e.g. DiRuffano et al., 2017; Hoffmann et al., 2014) making it difficult to identify its active ingredi- ents; (2) the delivery of the intervention limiting the understanding of its specific and nonspecific components; and (3) the mechanism through which the interven- tion (including its specific and nonspecific components) affects the health problem constraining the ability to discern what aspects of the problem are addressed and how the desired changes are achieved (Abraham et al., 2014). Accordingly, the empirical approach assists in identifying interventions, more so than delineating their active ingredients. However, new methods have been used to compile evi- dence on the mechanisms underlying the intervention’s effects (e.g. realist review of the literature) and the active components comprising the intervention (e.g.
component analysis). Traditional and new methods for reviewing empirical evi- dence are discussed next.
4.2.2.2 Methods
General Review Process
The traditional and new methods for reviewing empirical evidence involve a search of the literature, critical analysis of the studies’ reports, and synthesis of the studies’
findings. The two types of methods share similar steps to: search the literature; select reports for review; and extract information. They differ in the analysis and synthesis of the extracted data and studies’ findings.
Literature Search
The search is done in various bibliographic databases (general and specific health- related disciplines) for comprehensiveness. The search uses keywords that capture the health problem, its indicators, and each of its determinants judged as potentially modifiable. The keywords are combined with those reflecting possible interventions or techniques, including: (1) the name/label of generic intervention techniques (e.g.
behavioral intervention) or specific therapies or methods (e.g. persuasion), which may have been suggested and/or actually used to address the modifiable aspects of the problem, or (2) words that are synonymous with the term “intervention” such as therapy, technique, method, care, treatment, service, and program. The search may be limited to a recent time period (e.g. past 10 years) to enhance the relevance of interventions to the current context.
Selection of Reports
The selection of reports is based on pre-specified criteria. The criteria are related to:
(1) the health problem or its aspects addressed by the intervention under evaluation;
(2) the characteristics of the client population (e.g. age or gender) and context (e.g. rural areas) of interest; (3) the setting in which the intervention is delivered (e.g.
client’s home); and (4) the study design (e.g. randomized clinical trials). It may be
useful to specify broad criteria (in particular those related to study design) to enhance comprehensiveness of the evidence. In addition to reports of individual studies, those presenting findings of literature reviews (e.g. scoping, narrative, systematic) and/or describing a study protocol are selected because they provide information on the effectiveness and on the nature of the intervention and its components, respectively.
Data Extraction and Analysis
The information to extract from the selected reports of studies is presented in Table 4.1. It covers methodological and substantive characteristics of the study. Of importance in informing the delineation of the intervention’s active ingredients are details on the intervention related to its components, mode and dose of delivery, mechanism of action, and effectiveness in inducing the desired changes in aspects of the problem and in improving the experience of the problem. The analysis and syn- thesis of the extracted data vary by review methods.
Traditional Methods
The traditional review methods include scoping reviews, systematic reviews, and meta-analyses. Scoping reviews involve a comprehensive search of the literature to identify the types of interventions that have been proposed, used, and/or found
TABLE 4.1 Information to extract from selected reports of empirical studies.
Category Examples
Methodological
characteristics Type of design
Number and type of study groups Time points for outcome assessment Sample size
Quality of study Intervention
characteristics Goal of intervention
Active ingredients, or specific and nonspecific components Content covered or activities in which clients or participants
engage Mode of delivery Dose
Type and training of personnel or staff involved in intervention delivery
Hypothesized
outcomes Type of outcomes
Measures used
Results Findings quantifying the intervention’s effects on the hypothesized outcomes
Additional findings related to:
1. Fidelity with which the intervention is delivered 2. Participants’ exposure (number of sessions attended),
adherence (application of treatment recommendations), and satisfaction with the intervention
3. Factors that influenced the delivery and/or outcomes of the intervention
4. Mechanism underlying the intervention’s effects (either tested empirically or perceived by participants and/or staff) 5. Unexpected outcomes
4.2 Approaches for Delineating the Intervention’s Active Ingredients
75
useful to address the health problem or its malleable aspects. For example, Guruge et al. (2017) conducted a scoping review to map out interventions addressing stigma of mental health. Systematic reviews and meta-analyses focus on determining the extent to which interventions are effective in improving the health problem. In all types of reviews, content analysis of the selected reports is applied to identify qualitatively similar interventions. Codes are assigned to different categories of inter- vention (e.g. medication, education, behavioral).
In systematic reviews, vote counting is used to synthesize the evidence on the interventions’ effects. This consists of reviewing the findings of each study to deter- mine if the interventions produce: no significant effects; significant effects in the hypothesized direction (i.e. clients or participants exposed to the intervention report improvement in the problem); or significant effects in the non-hypothesized direction (i.e. participants exposed to the intervention reported worsening in the problem).
Interventions are considered effective if they produce significant effects in the hypothesized direction in most (more than 50%) studies included in the review (Hong et al., 2017).
In meta-analyses, statistical methods are applied to estimate the effect size, which quantifies the magnitude of the intervention effect on each outcome exam- ined in each study included in the review. The extent of variability in the effect sizes is examined across studies. Low variability suggests that the intervention’s effects on the outcomes are consistent; in this case, the effect sizes are aggregated across studies, taking sample size into account, and yielding an average point estimate (with confidence interval) of the intervention’s effects. High variability prompts the inves- tigation of possible causes of the heterogeneity in the estimated effect sizes. Possible causes are characteristics of the client population (e.g. gender, age), the intervention (e.g. components, mode or dose of delivery), the context in which the intervention is delivered (e.g. personnel providing it, setting), or the study (e.g. design, outcome measure). Advanced statistical tests (e.g. meta-regression) are used to relate the effect sizes to possible causes (Hong et al., 2017; Paré et al., 2015). The findings may point to variability in the effectiveness of different interventions in different client population and contexts. Such empirical evidence is most informative in selecting interventions as it indicates the intervention or the components that are most successful in the same or similar client population and context of interest.
New Methods
New methods for reviewing empirical evidence include mixed methods reviews or syntheses that aim to delineate the interventions’ active ingredients and mechanism of action. To address the first aim, intervention component analysis is proposed and illustrated in the Distillation and Matching Model (Chorpita et al., 2005). The inter- vention component analysis consists of several steps: (1) review relevant theory- based interventions to identify their active ingredients and their operationalization into components and techniques; (2) develop a guideline for coding them, where the guideline specifies the name of the components or techniques and describes what they entail; (3) search and select empirical studies that evaluated the interventions, components, or techniques; (4) extract descriptions of the interventions and results of their evaluation; (5) review the extracted descriptions of interventions and code for the presence of components or techniques; (6) generate matrices that summarize the coded component or techniques and the results of evaluation studies; and (7) analyze the data, using appropriate algorithms or statistical tests to explore pat- terns in the matrices that indicate which combination of theoretically derived
components or techniques is associated with beneficial outcomes. A variant of this method was applied by Fox et al. (2013) to identify the most effective components of the acute care for elder (ACE) model.
To address the second aim, mixed methods review and synthesis are recom- mended. The review focuses on delineating the interventions’ mechanism of action.
It involves selection and review of studies that evaluated the process of implementing and of the outcomes of interventions. Process evaluation (see Chapter 13) involves the investigation of the fidelity with which the intervention is delivered, of contex- tual factors affecting the delivery and effectiveness of the intervention, and of clients’
perception of the intervention and its impact. Outcome evaluation (see Chapters 14 and 15) is concerned with examining the effectiveness of the intervention in inducing the desired changes in aspects of the health problem (referred to as immediate and intermediate outcomes) and improving the experience of the health problem and preventing its consequences (posited as ultimate outcomes).
The quantitative and qualitative findings pertaining to process and outcome are analyzed separately or concurrently, and integrated to delineate the pathway linking the intervention or its components and techniques to the immediate and intermediate (i.e. desired changes) and the improvement in the ultimate outcomes (i.e. health problem experience) (Edwards & Kaimal, 2016; Gough, 2013; Snilstveit et al., 2012;
White, 2018). Examples of mixed methods review are framework synthesis (Carroll et al., 2013) and realist reviews (Pawson et al., 2005). Both involve (1) engagement in an initial exercise to clarify the problem and to model the intervention’s mechanism of action and contextual factors that may affect the delivery, mechanism of action, and/or outcomes of the intervention; this initial model is represented in a configura- tion that links context, mechanism, and outcome of the intervention; (2) review of the theoretical literature and empirical (quantitative and qualitative) evidence and extraction of data reflective of contextual factors, intervention components or tech- niques, desired changes and effectiveness in addressing the health problem; (3) cod- ing of the extracted data relative to configuration of context, mechanism, and outcome; and (4) analysis and integration of the theoretical and empirical evidence to determine the adequacy or the need to revise the initial model. The application of the realist review is illustrated in the work of Robert et al. (2017).
4.2.2.3 Strengths
The empirical approach has the advantage of identifying interventions that have been implemented and evaluated. The evidence suggests that the interventions are feasible and effective in preventing, managing, or resolving the health problem, in different client populations and contexts.
4.2.2.4 Limitations
Extant empirical evidence has limitations: (1) the interventions are often poorly or briefly described, which creates difficulty in appropriately coding them, and the potential of lumping variant interventions in the same category; variability within categories may yield underestimated effects for the categorized interventions;
(2) there is a rather small number of studies that evaluated the effects of different active ingredients or components comprising an intervention, the fidelity of inter- vention delivery, and the context and mechanism underlying the intervention’s effects; yet this evidence is useful to delineate the intervention’s active ingredients
4.2 Approaches for Delineating the Intervention’s Active Ingredients
77
and to select the components that are most effective in addressing the potentially modifiable aspects of the health problem; (3) there is growing acknowledgement of publication bias, which threatens the validity of the synthesized empirical evidence;
that is, there is a tendency to selectively report positive outcomes (Chan et al., 2014) and publish reports of studies with “statistically significant” effects; when averaged, these effects yield overestimated intervention’s effects (van Assen et al., 2015).
The application of the theoretical and empirical approaches requires the avail- ability of relevant theory and theory-based therapies or techniques, and evidence respectively. In situations where theory and evidence are not accessible, the experien- tial approach is advocated, whereas in situations where theory has not been tested and evidence has not been generated from studies involving the client population and the context of interest, the combined approach is recommended.