Modeling
5. ISSUES TO CONSIDER IN BUILDING A POLICY ANALYSIS MODEL
Conducting a policy analysis may not require you to build detailed models of the phenomena you seek to study; indeed, in many cases, your job will be primarily to use existing models and their results. Nevertheless, as you synthesize information from various sources, the modeling issues described below will help you to put all the information together in a consistent, well justified analysis.
5.1 Problem Definition
Chapter 1 has already discussed the importance of problem definition, and Chapter 2 has helped you think of the time, budget, and other constraints that you face in conducting your analysis. The following issues must be addressed in building a model. You need to ensure:
(a) that your model answers the questions that you need answered. For instance, if you need the costs of an alternative, a purely ecological model will not serve your purposes.
(b) that you will have sufficient time, money, and expertise available to build a model sufficient for your needs. Modeling can be very expensive and time-consuming, or a combination of rules of thumb and existing research can be pulled together fairly rapidly and cheaply. Either approach may be appropriate for the purposes of you and your client, as long as you and your client have negotiated these aspects in advance.
(c) that you have thought about both the current use of the model and possible future uses. If you expect that your model will be used only once, your model can be made highly specific to the conditions of your current study. If you think your model will be used again in the future, you will want to build it so that it can be applied to multiple situations.
(d) that you have thought about the consequences if your model is wrong.
If your model is assessing impacts of a major, expensive activity with possibly irreversible environmental and social effects, then you are likely to want to
spend more time and effort on your model to make it as accurate as possible. If the consequences of the model predictions are reversible (for instance, farmland might be brought out of production temporarily), then the need for being precise is much smaller, and you (and your client) could save money on the modeling effort.
5.2 Model Scale
At what level should your model be developed? Will it be applicable to an individual, a neighborhood, a city, a county, a region, a state, or nation, or the world? Will it work for some pollutants but not others? Will it work for water pollution but not air pollution? Will it apply to all ruminants, or only to deer?
A model built for one scale may not work very well when applied to a different scale.
If you are using an existing model rather than building one, this issue can be especially important. Does the model you propose to use apply to your situation? You may want to do some validation of the model to determine if it is appropriate for your project.
5.3 Formulation of the Model
The assumptions that you plan to use in modeling need to be made explicit.
As discussed above, you will want to draw on theoretical literature to determine what variables are important, what relationships these variables have to each other, and when this model will be applicable.
5.4 Data Collection
Earlier in this process you identified what variables are important.
Sometimes you will need to collect original data using surveys or interviews; in other cases, you can rely on existing data sets. In either case, you need to get data that match what you need as closely as possible. Like model building, data collection can be time-, expertise-, and budget-intensive, or it can be done with very little of any of those.
The quality of data depends not only on the scientific rigor with which it was assembled, but also its applicability to your question. If you need data on how elk respond to timber harvesting, will data on deer, no matter how carefully collected, really serve your purpose? If you need household income for a city, will per-capita income for the state be an appropriate proxy? Again, you need to consider scoping questions as you collect your data.
5.5 Model Estimation or Calibration
For a statistical model, this is the point where you will use the data to estimate the coefficients in the model and/or apply the model coefficients to make predictions. For a simulation or programming model, you will assemble the equations and their relationships to each other and run the computer simulations. Considerations such as whether linear or non-linear functional forms are superior will often be tested at this point. In any case, you want your model to be the “best fit” for your data that you can achieve.
5.6 Model Validation
How do you know if your model does what it is supposed to do? Models are usually validated by running them for situations with known outcomes, and comparing the model's results with what actually happened. If you cannot conduct validation of this sort (due, for instance, to lack of data for comparison), peer review of your model can give you useful feedback. For multiple regressions, a variety of "goodness of fit"tests can be performed as well (see Chapter 6).
The results of your validation efforts are almost certainly not going to perfectly replicate the real outcomes; how similar they should be depends on how precise you need your model to be (Step 1). In some cases you will not be satisfied unless they are almost the same; in other cases, if the results are of the same order of magnitude, or even the same direction, you will be satisfied. If your model results do not satisfy you, you will need to check your assumptions, your data, and your model estimation to try to figure out where errors lie.
If you have concerns about some of your assumptions or some of your data, part of your model validation could include sensitivity analysis of these factors. If the results do not change much, the model is said to be robust to the different assumptions. If the results do change more than you desire, then you should probably put more time and effort into ensuring that you have these factors as correct as possible.
Once your model is performing to your satisfaction, you can now use it for the functions for which you developed the model. Most likely, you will use your model to predict the consequences of various alternatives to fill in part of a decision matrix (see Chapter 3) or as a decision rule to select the preferred alternative.