Usefulness of Sampling Plans 6 6
6.4 Evaluation by Users, Field Tests and Simulation
present the decision guide as a graphic and to include numerical values with the graphic. This is illustrated in Fig. 6.1. The benefit of a graphic is that it helps users to understand the logic of what they are doing during sampling. The benefit of including numerical values for stop boundaries is that interpolation is simplified.
In the previous sections a lot of emphasis has been put on meeting practitioners’
needs when developing decision guides. This might suggest that development of decision guides is predominantly a customer satisfaction enterprise, in which devel- opers of decision tools should develop what practitioners like to use. We do not intend to give this impression. There is a large body of knowledge about pest sam- pling that can be put to good use by smart designers who take account of the man- agement objectives of practitioners, and who have the insight and tools to create decision guides suited to specific needs. The simulation tools presented in previous chapters allow a smart designer to develop sampling plans that may be unfamiliar
to the client, but which provide satisfactory management advice (as described in the OC function), and yet keep the sampling effort at as low a level as possible (as described by the ASN function). In a fruitful collaboration, practitioners will ask questions and developers will provide answers, as well as the other way round. This produces a continuous cooperative evaluation process in which all participants learn and teach.
What, if any, is the role of field experiments in the evaluation process? One reason why field tests can be viewed as indispensable is that the models used to represent sample observations may be inadequate and may not cover the full variability of real systems. If this were the case, estimates of OC and ASN functions calculated by the tools described in previous chapters would not be representative of what happens in the real world. However, this should not be a critical concern, because OC and ASN functions are relatively robust to departures from the models used to represent sample observations (Nyrop et al., 1999). Therefore, field experiments are not essential for estimating OC and ASN functions. This is indeed fortunate, because field experiments cannot realistically be used to estimate OC and ASN functions directly. This is because in order to estimate a point on the OC and ASN functions, the sampling plan must be applied many times to populations with the same or similar means, and this process must be repeated over a range of means.
This would be extremely difficult, if not impossible, to accomplish in the field. As an aside, we note that field data can be used in a simulation context to estimate OC and ASN functions; we will return to this in Chapter 9.
While field experiments can generally not be used to estimate OC and ASN functions, they do serve three important roles in the development and evaluation of crop protection decision guides. First, they can provide real-world estimates of the actual time and resources required to obtain sample information. Second, they can alert developers to possible erroneous assumptions or factors that they may have overlooked in the development process. These assumptions or factors are not related to the distribution of the sample observations, but to the timing of sampling during the day, the selection of sample units and other types of bias that may influ- ence the sample counts. Finally, field experiments can serve to demonstrate to prac- titioners the usefulness of the decision guide.
The simplest way to test a guide using a field experiment is to apply the sample protocol at a site and follow it immediately with validation sampling, to provide a precise and accurate estimate of pest density or incidence. If the influence of pest phenology on sample outcome is being evaluated, the timing of the validation sam- pling must be carefully considered. By comparing the results from the decision guide with the validation estimate, one can determine whether the guide provided the correct advice. The decision guide might also be used by several testers at the same site to provide information on possible observer bias. Another possibility is to test the sampling plan over a short period of time during which the actual pest
Enhancing and Evaluating the Usefulness of Sampling Plans 139
6.4.1 Field evaluation
density does not change, but the conditions (e.g. light, wind and temperature) under which the sampling plan is used do change.
We have stated that OC and ASN functions are relatively robust to variability in the models used to represent sample observations. Given this robustness, we believe that a viable strategy for developing sampling methods for a new crop–pest system is to guess the parameters used to describe the distribution of sample counts on the basis of similar crop–pest systems, and perform a sensitivity analysis (see, e.g.
Exhibits 5.1–5.3) to see whether the performance of a pest-control decision guide would be sensitive to any differences in parameter values in the plausible range (Nyrop et al., 1999). This is not a novel insight. Green (1970) and Jones (1990) came to similar conclusions, although from the perspective of estimating abun- dance rather than from the view of sampling for decision-making. Elliott et al.
(1994) also suggested that generic models of sampling distributions might suffice for developing sampling plans for pests of wheat and rice. From the perspective of sampling for decision-making, we think an even stronger case can be made for the use of generic models of sampling distributions.
Based on work described in the literature and the examples provided thus far in this book, we make three recommendations for developing sampling plans for use in pest management decision-making.
Ideally, the first step is to obtain a good estimate of the critical density or critical proportion (cdor cp).Of course, this may not be possible, and it is often the case that using some value for this parameter, even though not the theoretically correct value, would markedly improve pest management decision-making, compared to cal- endar- or phenology-based tactics. It is then important to recognize the vagueness of cdor cp, and to understand that precise information about pest abundance pro- vides a false sense of security. In such cases, reductions in decision errors will come from improved knowledge of cpor cd, rather than from increasing the precision of the sample information.
The second recommendation is that when developing sampling plans, one should start by determining what is known about the sampling distribution, use this infor- mation to develop a sampling plan, and then perform a sensitivity analysis to deter- mine if further refinement of this information is warranted. When nothing is known about the sampling distribution for a particular pest, sampling distribution models might be grouped for similar taxa on similar crops (e.g. aphids on small grains, or disease incidence on perennial fruit crops). This hypothesis is easily tested using simulation.
6.4.2.2 Sensitivity with respect to sampling distribution 6.4.2 Simulation, sensitivity and robustness
6.4.2.1 Critical density
The third recommendation is that sample size should first be based on the require- ment that the sample information should be representative. This means that the sample information must be a good indicator of pest abundance or incidence in the management unit. Sample information may not be representative if sample obser- vations are not collected from throughout the sample universe, which can happen when very few samples are taken. Therefore, when using sequential sampling, a rea- sonable minimum sample size and sample path need to be defined.
A final important evaluation question to answer is whether collection of the sample information leads to reduced pest control costs. This is addressed in the next section.
What is the value of using decision guide sampling to recommend a management action, as against always intervening, or as against always leaving nature to take its course? At one extreme, farmers whose crops rarely have pest problems might feel that spending time and money collecting samples to tell them mostly what they already know (don’t intervene) would be wasteful. At the other extreme, farmers whose crops nearly always require treatment might feel the same – and continue to treat. For these two extremes, and between them, we can calculate the long-term economic value of decision guide sampling. The results provide a comprehensive economic scale for comparing decision guides.
The economic value of the sample information used to guide pest management decision-making is the reduction in pest control costs that can be attributed to basing management decisions on the sample data. While it is desirable that pest control costs be reduced as a result of using the sample information, a sampling plan with a negative economic gain may still be worthwhile if it carries other bene- fits, such as reduced pesticide inputs (Nyrop et al., 1989). We use the approach for quantifying the economic benefit of sampling proposed by Nyrop et al. (1986), which is rooted in economic decision theory.
To quantify the economic benefit, we require more detailed information about the context in which decisions are made than is needed to estimate OC and ASN functions, although these functions are still used in the analysis. First, we need a quantitative relationship between pest density, µ, and monetary loss when no con- trol is used, L(µ). In theory, such a relationship is used to provide an initial esti- mate of the critical density (see Chapter 1), but a provisional critical density can be formulated when the pest–loss relationship is rather poorly defined. Second, we need to know the likelihood of encountering each pest density within the range of possible densities. A probability distribution function is used to model this informa- tion. Because it provides the likelihood of any pest density before a sample is taken, it is called the prior distribution, P(µ), of the pest density, µ. Another way of look- ing at this prior distribution is to think of it as a long-term frequency of occurrence for each pest density.
Enhancing and Evaluating the Usefulness of Sampling Plans 141
6.4.2.3 Size and representativeness of sample