w
Understanding and Applying Economic Evaluation and Other Analytic Tools
There are in fact two things: science and opinion. One begets knowledge, the latter ignorance.
Hippocrates
T
he preceding chapters have underlined the desirability of using evidence to inform decision making in public health. The first chapter gave an over- view and definitions of evidence- based practice. The second chapter made the case for expanding capacity for evidence-based public health. The third chapter described the scientific factors to consider when determining whether some type of public health action is warranted. This chapter describes several useful tools for evidence- based public health practice that help practitioners answer the question, “Is this program or policy worth doing?” The primary focus is on economic evaluation, which compares the costs and benefits of a program or policy as one way to address this question. Six other analytic tools are also presented. Epidemiology, which is its own area of analytics, is presented in the seventh chapter.Chapter 4 has five main parts. First, we describe some context for these methods. Then we describe economic evaluation, a set of methods for com- paring benefits and costs. One particular type of economic evaluation, cost- effectiveness analysis (CEA), is described in greater detail. The third part discusses several analytic tools for measuring intervention impact and effec- tiveness. In the fourth section, several challenges and opportunities in using these analytic tools are discussed. A major goal of this chapter is to help read- ers develop an understanding of these evidence- based methods and an appre- ciation of their usefulness. The number of publications using these methods, particularly economic evaluation, has grown exponentially over the years.
( 76 ) Evidence-Based Public Health
We seek to assist practitioners in becoming informed consumers of these publications.
BACKGROUND
Economic evaluation aims at improving the allocation of scarce resources.
Given that we cannot afford to do everything, how do we choose among projects? Economic evaluation identifies and weighs the relative costs and benefits of competing alternatives so that the project with the least costs for a given benefit, or the greatest benefits for a given cost, can be identi- fied and chosen. Economic evaluations can be conducted before implemen- tation of an intervention to assess feasibility, alongside interventions,1- 5 or after the intervention has concluded. Economic evaluations can use pro- spective data to determine the cost- effectiveness of a new project, use the existing literature to forecast the impact of a proposed program or policy, or use a combination of both prospective data and the existing literature.
The number of economic evaluations has grown over the years, and public health decision makers can now search the literature for economic evalua- tions of a potential intervention to help them decide whether to undertake that intervention.
Several quantitative techniques help support economic evaluations. For example, a key component in economic evaluation is the cost of the illness or disease that the intervention is designed to address. Resources will be saved by preventing the condition or treating it more effectively. Cost of illness stud- ies measure the direct and indirect costs of diseases and conditions, giving an estimate of the anticipated number of individuals experiencing the condi- tion and the potential costs saved by preventing the illness or disease and its sequelae.
Another useful quantitative method for economic evaluation is decision analysis. A necessary step in all economic evaluations is the specification of all alternatives and their costs and benefits. For example, some participants in a smoking cessation program may quit smoking, some may quit smoking and relapse, and some may continue to smoke. These groups may then differ in their probability of having lung cancer later in life. This part of the analysis can be complex, with multiple alternatives, each with their own probability of occurrence. Decision analysis is an analytic tool designed to assist with com- plex decision making.
Finally, only effective interventions should be assessed with economic eval- uation, so knowing the effectiveness of interventions is key. Several quantita- tive methods are relevant to determining the effectiveness of an intervention.
Meta- analysis and pooled analysis are two methods to quantitatively combine
Economic Evaluat ion and ot hEr analy t ic tool s ( 77 )
the results of multiple studies. Risk assessment is a method to determine the risks to people due to an adverse exposure. Health impact assessments are a related method that determines the risk and benefits to health posed by interventions commonly thought to be non– health related, such as subsidized housing or public transportation.
ECONOMIC EVALUATION: A TOOL FOR COMPARING OPTIONS AND WEIGHING BENEFITS VERSUS COSTS Economic evaluation is the comparison of costs and benefits to determine the most efficient allocation of scarce resources. We undertake economic evalu- ations all the time in everyday life, though we seldom think of the process explicitly. For example, ordering lunch at a fast- food restaurant requires weighing the costs (monetary and caloric) versus the benefits (nutrition and flavor) of all of the options. Then, we choose a meal that is the “best” use of our resources— the best value for the money. This implicit weighing of costs and benefits is almost automatic, though we’ve probably all faced a menu that seemed to offer too many options at one time or another. In most public health applications, however, weighing the costs and benefits does not hap- pen so automatically.
What are the distinguishing features of public health that require a for- mal economic evaluation? Consider three features of the restaurant example.
First, the costs and benefits are all borne by one person, the diner, who has an incentive to compare costs and benefits and make a wise choice. Second, the information needed for the choice is fairly easy to obtain. In many fast- food restaurants, the food choices are described and listed along with the prices and calories. The diner knows his or her own palate and preferences. Finally, the stakes are fairly low. A bad decision can be remedied by ordering another item (though this has costs) or by avoiding the food choice or restaurant the next time the diner eats out.
All three of these characteristics are absent from most public health deci- sions. First, by their nature, public health programs are aimed at improving the health of a community, so benefits will be spread over a large number of people. Costs are also typically spread over a large group, often through taxation. Second, the information about costs and benefits may not be easy to obtain. Benefits and costs must be measured over many people. Often, the benefits include hard- to- measure items like improved health status. Third, the stakes are often relatively high. Programs may be expensive and resources scarce, so only a few of a large range of interventions may be funded. A bad choice cannot easily be remedied.
( 78 ) Evidence-Based Public Health Types of Economic Evaluation
There are four interrelated types of economic evaluation: cost- benefit analysis (CBA), cost- effectiveness analysis (CEA), cost- utility analysis (CUA), and cost- minimization analysis (CMA). This chapter explains these methods, focusing primarily on CEA and CUA. These are the recommended methods of the US Public Health Service Panel on Cost- Effectiveness in Health and Medicine6 and the most commonly used methods today.
The four methods differ primarily in the way that they measure benefits.
CBA measures benefits in monetary units (e.g., dollars, Euros), whereas CEA measures benefits in a naturally occurring health unit (e.g., lives saved, years of life saved). CUA is a type of CEA in which benefits are adjusted for quality of life and quantified with a health utility measure (usually quality- adjusted life- years, or QALYs). CMA is only used when the benefits of the two inter- ventions are identical, so the unit of measurement of benefits is not an issue.
Because CBA uses the most “generic” outcome measure (many things can be measured in currency, including the value of transportation projects and edu- cational interventions), it allows for the comparison of the most programs. As we move to CEA, then to CUA, and finally to CMA, the range of programs that can be compared narrows.
Economic evaluation is closely related to return- on- investment (ROI) anal- ysis, business plans, and capital investment decision tools. All of these busi- ness tools are undertaken within businesses to compare the costs and benefits of a proposed project or investment to the business entity. As in the fast- food restaurant example, these methods assume that all of the costs are borne by the business and all of the benefits accrue to the business. Economic evalua- tions use many of the same analytic methods but differ by including costs and benefits that accrue to multiple parties, such as the individual, his or her social network, and society.
All economic evaluations compare one intervention or program to another.
Figure 4.1 provides a diagram of an economic evaluation. As shown in the fig- ure, the alternative program can be “standard care” or “no program,” but the analysis is always framed as, “What are the extra, or incremental, costs and the extra, or incremental, benefits of this program compared with another program?” This requirement is helpful to the analyst because it is often easier to determine the incremental costs and benefits than the total costs and ben- efits. The primary outcome of an economic evaluation is the incremental cost- effectiveness ratio (ICER):
ICER= Incrementalcosts Incremental benefits
The particular items included in the numerator and denominator will depend on the intervention and the type of economic evaluation.
Economic Evaluat ion and ot hEr analy t ic tool s ( 79 )
Outcomes of an Economic Evaluation
Figure 4.2 shows the potential outcomes of an economic evaluation.7 Consider the four quadrants of the graph. Programs that improve health and save money (quadrant IV) are obviously worthwhile and should be undertaken. Similarly, programs that worsen health and add to costs (quadrant II) are undesirable and should not be initiated or continued. The remaining two quadrants (I and III) are where the dilemmas lie and where economic evaluation can be informative.
Historically, as public health systems develop, interventions and pro- grams begin in quadrant IV, with those programs that are both cost saving and improve health. Many early public health interventions, such as sanita- tion systems, fall in quadrant IV. As more of these interventions are imple- mented, attention turns to quadrant I, programs that improve health at some cost. In times of budgetary pressures, quadrant III programs are consid- ered: programs that reduce costs, but at some loss of health status. For both of these quadrants, the question is, “What is the return on the investment (or
Choice
Program B
Comparison program May be new or old Could be ‘doing nothing’
Track inputs Track outputs
New Program A Benefits from A
Benefits from B
Costs
Direct Indirect
Averted Treatment Costs
Benefits
YOLS QALYs Dollars
Figure 4.1: Diagram of an economic evaluation. QALYs = quality- adjusted life- years; YOLS
= years of life saved.
Quadrant IV Quadrant I
Quadrant II Quadrant III
Saves money,
Improves health Costs money, Improves health Costs money, Worsens health Saves money,
Worsens health
Aggregate Costs Aggregate Health Benefits
Figure 4.2: Possible outcomes of an economic evaluation.
Source: Adapted from Drummond et al.7
( 80 ) Evidence-Based Public Health
disinvestment) of the public’s funds?” Economic evaluation provides a way to answer this question so that programs with the greatest return on investment can be selected. For example, consider quadrant I. Suppose that two interven- tions are considered. Plotting the incremental costs and benefits of the two interventions yields one point that is to the northwest of the other point.
Comparatively, that intervention dominates the other intervention because it has lower incremental costs and higher incremental benefits. This is an exam- ple in which economic evaluation can help choose between two interventions, both of which improve health.
Common Steps of an Economic Evaluation
Whether one wants to conduct an economic evaluation or use the results of an existing evaluation, it is helpful to understand the mechanics of these analy- ses. Every economic evaluation includes nine steps:
1. Identify an effective intervention for the problem and people at risk 2. Select the perspective of the analysis
3. Select the type of economic evaluation to perform 4. Measure costs
5. Measure outcomes
6. Discount costs and benefits as needed 7. Construct the ICER
8. Conduct a sensitivity analysis
9. Compare the ICER to external standards or use internally
In this section, the first three steps are considered. The remaining steps are considered separately.
The first step is to identify the intervention and the group. Unless the economic evaluation is to be conducted alongside a new intervention, the intervention should have already been demonstrated to be effective. There is nothing to be gained from an economic evaluation of an ineffective inter- vention. The intervention and the group it applies to should be specified as completely as possible, including identifying the expected benefits of the program.
The second element is the selection of the perspective of the economic evaluation. Any intervention can be considered from several points of view, often characterized as moving from narrow to broad. The narrowest perspec- tive is that of the agency or organization directly involved in delivering the proposed intervention. A next level might be the perspective of insurers, or payers, especially in health, where consumers and payers are often two sepa- rate groups. The broadest perspective is that of society as a whole. The Panel
Economic Evaluat ion and ot hEr analy t ic tool s ( 81 )
on Cost- Effectiveness in Health and Medicine recommends this broad per- spective for all economic evaluations,6 and it is required in several countries with national health systems. The societal perspective is the most appropriate in public health because interventions are designed to benefit the public and taxpayers who fund the costs.
However, an analyst may wish to conduct an economic evaluation from two perspectives, or highlight some aspects of an analysis for particular stakehold- ers. For example, a workplace injury prevention program can be analyzed from a societal perspective, counting the avoided pain and suffering of workers as a major benefit. Another benefit to the employer is the avoided lost productiv- ity and avoided medical claims. Highlighting these benefits for the employer may encourage employer participation and may even justify some payment for the program by the employer.
The perspective of the analysis will determine which costs and benefits are included in the analysis. In general, the broader the perspective, the more costs and benefits that are included. This chapter focuses on the societal perspec- tive and thus lists all of the possible costs and benefits that can be included.
If a narrower perspective is taken, some of the costs or benefits in the text and tables may not be counted. The perspective can be thought of as a fishing net— the broader the perspective, the larger the net that is cast, and the more that is “caught,” or included.
The third step is the selection of the appropriate type of economic evalu- ation. Table 4.1 shows the different types of economic evaluation and their defining characteristics.8- 10
Selection of the type of economic evaluation primarily depends on the ben- efits of the program. If they can be easily measured in monetary units, then CBA is appropriate. If there is a primary benefit that can be measured as a nat- urally occurring health unit, then CEA can be used. If there are multiple health benefits, such as mortality and morbidity reductions, then CUA is appropri- ate. Finally, if the benefits of the programs being compared are identical, then a CMA can be used.
Measure Costs
The fourth step is the identification and measurement of all incremental costs of a program, option, or intervention. Incremental costs are the additional costs related to the program. The scope of the costs is determined by the per- spective of the analysis. If such costs are concentrated among a small group of people, this step will be relatively easy. As costs are more dispersed, it may become more difficult to identify all potential costs. Measurement of the iden- tified costs may similarly be complicated by issues of units of measurement (e.g., monetary wages vs. donated labor time) and timing (e.g., costs incurred over a 5- year interval).
Table 4.1. TYPES OF ECONOMIC EVALUATIONS AND MEASUREMENT OF BENEFITS
Type of Analysis Benefit (Outcome) and Example Measurement of Benefits Cost- minimization
analysis (CMA)
Identical, but costs are different Can we do it for less?
CMA compares the costs of different programs that produce the same health- related outcomes.
Example:
Program A. Participants walk 4 days per week.
• Lowers cardiovascular disease (CVD) risk by 10%
• Costs $3000 per participant per year Program B. Participants reduce fat from 40% to 30% of calories
• Lowers CVD risk by 10%
• Costs $2500 per participant per year Both programs are equally effective in lowering CVD risk, so compare data on costs only.
Decision: Choose the reduced fat intake intervention
None (identical measure and amount of benefit)
Cost- benefit analysis (CBA)
Single or multiple benefits (outcomes) standardized into a single monetary value (in present dollars)
Is there a reasonable return on investment?
CBA compares the costs and benefits of 2 or more programs using monetary outcomes. It is the gold standard for economic evaluation (EE) and is the most common form of EE in business (also called return- on- investment [ROI] analysis).
Lower cost- benefit ratios and higher net benefits are desirable. Ratios <1 indicate that the program is cost saving.
Example:8
Intervention: Neighborhood- based program to prevent teen pregnancy
Program costs: $9,386 per participant per year Effects: reduced teen pregnancy from 94/ 1000 to 40/ 1000
When combined with effects, the cost per birth averted is $26,142
The savings per birth averted (or benefits) are
$81,256
The incremental cost- effectiveness ratio (ICER) for the program is $26,142/ $81,256, or 0.32. The net benefits are $55,114 (=$81,256 – $26,142) per birth averted
Monetary units (e.g., dollars, Euros)
Type of Analysis Benefit (Outcome) and Example Measurement of Benefits Cost- effectiveness
analysis (CEA)
Single common benefit (or outcome) Are the (natural) outcomes worth the cost?
CEA compares the costs and benefits of different programs using the same outcome measure.
Outcome measures are naturally occurring health outcomes, such as cases found, years of life saved, or injuries prevented. CEA is easy to understand in the health field and avoids converting health outcomes to dollars. It is limited in its ability to compare interventions because those compared must have the same outcome. The result of a CEA (its ICER) is the cost per unit of health outcome (e.g., cost per year of life saved). Lower ratios are preferred.
Example:9
Intervention: Smoking cessation program in the workplace
Costs measured: All costs of the cessation program for all 100 participants— $8940 Effect measured: Number of people who quit smoking (quitters)— 15 quitters
The incremental cost- effectiveness ratio (ICER) = $596 per additional quitter
Natural units, e.g. life years gained, lower A1C levels, improved physical activity
Cost- utility analysis (CUA)
1 or more benefits (outcomes) standardized into a single value
Are standardized outcomes worth the cost?
CUA compares the costs and benefits of a program, with benefits measured in health- related quality of life- years (QALYs). CUA allows for comparison of many projects with health- related outcomes and is useful when both morbidity and mortality are affected or the programs have a wide range of outcomes but all have an effect on healthy years of life. Translating health outcomes, particularly morbidity, to years of healthy life is controversial. The result of a CUA (its ICER) is the cost per QALY ($/ QALY). Lower values are preferred.
Example10:
Intervention: Diabetes self- management programs in primary care settings
Program costs: $866 per participant per year, total lifetime costs of program $11,760
Several effects: 87.5% benefited, A1c - 0.5%, total cholesterol. – 10%
Using QALYs to add up all effects results in lifetime gain of 0.2972 QALYs
ICER: $39,563/ QALY saved
QALYs Table 4.1. CONTINUED