The Operating Characteristic Function

Concepts of Probability and Principles of Sampling

6.6 The Operating Characteristic Function

In Sect. 6.5, we described an n = 10, c = 2 sampling plan. If we are going to use this plan, we want to know what assurance it will give us that the plan would identify an unacceptable batch. In other words, how discriminating is the n = 10, c = 2 sampling plan? It is possible that, due to random effects in sampling, the plan will sometimes accept a poor lot if, by chance, we fail to sample positive units.

It is also possible for the plan to reject a good lot if, by chance, we happen to draw a higher proportion of positive units in the sample than is present in the lot as a whole. There is no way to avoid some degree of error, i.e., drawing a set of sample units that, by chance, are not perfectly representative of the lot, unless we test the entire lot. We can reduce the likelihood of such random sampling errors by

6.6 The Operating Characteristic Function

134

testing more sample units (larger n). In fact, we can reduce the risks of incorrect lot assessment to any desired level by making n sufficiently large. In practice, however, we usually have to seek a compro-mise between (a) large n (many sample units) and less chance of incorrectly classifying the batch, and (b) small n (few sample units) and larger chance of incorrectly classifying the batch as acceptable when, in fact, it is not, or rejecting a batch that is actually acceptable. The first error is described as the ‘consumer risk’, because the consequence is that consumers are exposed to a risk above that which is considered acceptable, while the latter error is described as ‘producer risk’ because the producer will be penalized despite that the product is of acceptable quality or safety. This is further discussed in Sect. 6.6.3. In the language of statistics the producers risk is a Type 1 error or a ‘false positive’, while the consumer’s risk is a Type 2 error or a ‘false negative’.

6.6.1 The OC-Curve

An operating characteristic function is used to describe the performance of a sampling plan. This is often depicted as an operating characteristic (OC) curve (Fig. 6.1). The horizontal axis shows a mea-sure of lot quality. One common meamea-sure of lot quality is the true proportion (or prevalence) of units in a lot that are defective, i.e., that do not conform to the criterion of acceptability (i.e., contain the target organism or have a count above some number m). This proportion is often designated p and can have values from 0 to 1 (or 0–100%). It should be emphasised that these are defectives in the sample unit, which generally will be different from the ‘serving size’ or ‘product unit size’. The actual serving size or product unit sizes are of more relevance for the estimation of risks to public health. Another measure of lot quality would be the mean concentration or mean log concentration in the lot (see Chap. 7).

The vertical scale of the OC plot gives the probability of acceptance, Pa for a given sampling plan characterised by n and c and, for a given true prevalence of defective units/samples Pa is the expected proportion of occasions that the results of testing according to the sampling plan will indicate that the lot is acceptable. In other words, Pa is an indication of the reliability of the sampling plan, or of our

Fig. 6.1 The operating characteristic curve for a sampling plan with n = 10, c = 2, i.e., the probability of accepting lots in relation to the proportion defective among the sample units comprised in the lot being examined. If, for example, the lot comprises 20% (p = 0.2) defective sample units, the lot will be accepted with a probability of 0.68 and rejected with a probability of 0.32

6 Concepts of Probability and Principles of Sampling

135

confidence that the sampling plan will not accept a lot that contains more than the proportion of defective units specified as being acceptable.

The probability of detecting an unacceptable batch (and, conversely, the probability of accepting it) is governed by the true prevalence of defective units. This is illustrated, using an n = 10, c = 2 sampling plan, in Table 6.1.

Pa is determined using the ‘Binomial Distribution’. The principal underlying the Binomial Distribution and, hence, the probability of acceptance are described in Sect. 6.6.2.

6.6.2 The Binomial Distribution

In probability theory and statistics, the binomial distribution describes the chance of finding a specific number of positives when drawing samples from a lot and applying a test that has only two possible outcomes, e.g., a positive or a negative. In microbiological testing for pathogens, in particular, this is frequently the case because it is hoped that the contaminant will be present only at low levels, if at all, and either is present in a sample unit or is not. However, a test that assesses whether a microbiological count is above or below some numerical limit m also gives either a positive or negative result. Pa can equally be determined using the binomial distribution.

To illustrate, consider, for example, 1000 chocolate bars of which 100 are contaminated with Salmonella. It might be expected that if one sample unit (bar) was chosen at random there would be a 1-in-10 chance that the sample unit contains a Salmonella cell. Conversely, there is a 90% chance that the contaminant would not be present in the sample unit. If two sample units are randomly drawn, each has a 1/10th chance of containing a Salmonella cell: the probability of not detecting Salmonella is 90% x 90%, i.e., 81%.¹ If three sample units are taken and a positive in any sample unit causes rejection, the probability of not detecting the contamination among three sample units is 73%, and so on. Since not detecting a Salmonella cell will lead to acceptance of the batch of chocolate bars, we can calculate the probability of acceptance as the product of the probability of not detecting a Salmonella cell in the total number of sample units tested, which can be expressed mathematically as:

1 In fact, because sample units are not returned to the lot after sampling, the probability of detection in any subsequent sample is slightly altered after each sample is taken. This is because the population size is slightly reduced after sam-pling. The probability of detecting a defective sample in this situation is better described by the Hypergeometric distri-bution. In practical situations, however, the difference in probability of acceptance due to this consideration is insignificant (since the amount of sample taken is negligible in comparison to the total lot) and calculations based on the Binomial distributions are adequate and lead to simpler calculations.

Table 6.1 Effect of true defective rate on the probability of lot acceptance using a 2 class attributes sampling plan

p (%) (true proportion of defective units)

Pa (probability of accepting the batch using an n = 10, c = 2 sampling plan)

0 1.00

10 0.93

20 0.68

30 0.38

40 0.17

50 0.05

60 0.01

6.6 The Operating Characteristic Function

136

P_a = −

(

¹ p1

)

^{× −}

(

¹ p2

)

^{×…× −}

(

¹ p_n

)

(6.1) where p1, 2, …. n is the true proportion of defective units in the lot (i.e., chance of finding a non-conform-ing sample unit) when 1, 2 …. n (respectively) sample units are tested. Since, in most practical situa-tions¹, p1 ≅ p² ≅ … pⁿ we can summarise the above equation as:

P_a = −

(

¹ p

)

^{n (6.2)}

where p is the true (or ‘just acceptable’) proportion of defective units in the batch and n is the number of sample units from the batch that are tested. Then Pa is the probability that the sampling plan would not detect a batch with greater than the acceptable frequency of defective units.

For sampling plans in which c > 0 somewhat more complex calculations are needed (Eq. 6.3) but all derive from the same principles of probability.

P n

c n c p p

a

c n c

= _!× −

(

^!

)

_! ^{× ×}

(

¹⁻

)

^{− (6.3)}

Equation 6.3 is called the Binomial Distribution and Eq. 6.2 is a special case of the Binomial Distribution for which c = 0. To use Eq. 6.3 to calculate the probability of acceptance, the cumulative form of the function:

P n

c n c p p

a i

c c n c

,

!

! !

Σ=

× −

( )

^{× ×}

(

⁻

)

=

∑

− 0

1 (6.4)

is used.

Computer spreadsheet software often includes built-in functions to automate these calculations.

For example, in Table 6.1, for the entry p = 20%, Pa = 0.68 and can be calculated using Eq. 6.4 or can be calculated in Microsoft® Excel using the function Binomdist(2, 10, 0.2, 1) in which the first num-ber (i.e., 2) is the numnum-ber of non-conforming units that can be tolerated for the batch to remain accept-able (the c value), the second number (i.e., 10) is the number of sample units to be tested (the n value), the third number (i.e., 0.2) is the true proportion of defective units in the batch (the p value) and ‘1’ is part of the syntax used in Microsoft® Excel to generate a cumulative probability (like in Eq. 6.4), rather than a probability density curve (like in Eq. 6.3).

Figure 6.1 shows the full operating characteristic curve for the n = 10, c = 2 sampling plan and highlights the probability of acceptance or rejection of a lot that has 20% defective units. For lots with fewer defective units the probability of acceptance using this plan is higher while for lots with a higher proportion of defective units the probability of acceptance is lower.

While we have described p here as the true proportion of defectives, to establish a sampling plan we now consider p not as the true proportion of defective units, but the maximum tolerable proportion of defective sample units. Adopting this approach, and by reference to the OC curve, the reliability of a given sampling plan can be evaluated. The greater the proportion of units in a lot that are defective, p, (i.e., that contain a pathogen or contain a level of microorganism above some specified level), the lower is the probability of acceptance (Pa) of that lot.

If, for example, we set an upper limit of 20% defectives (i.e., p = 20%), then using the n = 10, c = 2 sampling plan, the Pa would be 0.68 for a lot that has exactly that 20% defective units. This means that on 68 of every 100 occasions when we sample a lot containing 20% defectives, we may expect to have 2 or fewer of the 10 tests showing the presence of the organism, and thus calling for acceptance, while on 32 of every 100 occasions there will be 3 or more positives, leading to the lot being considered unacceptable. As noted, an n = 10, c = 2 sampling plan has a 68% probability of correctly identifying

6 Concepts of Probability and Principles of Sampling

137

a lot that has 20% or more defective units. If the true rate of defectives were 30% the probability of acceptance would decline to 38% (see Table 6.1). Similarly, if p = 10% such lots will be accepted 93 of 100 times, while if p = 40%, such lots will only be accepted 17 of 100 times. Thus, using an n = 10, c = 2 sampling plan, lots with 10% defective sample units will be accepted most of the time, but with 40% defective units, acceptance would be seldom. Note that in none of these situations does the n = 10, c = 2 sampling plan guarantee a correct result in all cases.

We might consider that a sampling plan that only provides 68% confidence of rejection of a lot with >20% defective units is not stringent enough. We can also use Eq. 6.2 to identify an alternative sampling plan (with c = 0) to determine how many samples would need to be tested to be confident, at some required level, that batches with greater than some specified proportion of defective units would be rejected, as follows:

P_a = −

(

¹ p

)

ⁿ

taking the logarithm of both sides:

log

( )

P_a ⁼n log

(

¹⁻p

)

and rearranging again:

n=^log

( )

P_a ^{/ log 1}

(

⁻p

)

By substituting in the required confidence of acceptance of a non-compliant batch (which is 1  − prob-ability of rejection of a non-compliant batch, i.e., 0.05 if we want to be 95% confident of rejecting a non-compliant batch), and the threshold of acceptability (which in this example is 20%), we find:

n=

( ) (

⁻

)

= −

( )

= − −

=

log . / log . . / log . . / . .

0 05 1 0 2 1 301 0 8 1 301 0 09691 13 443

i.e. 14 samples, none of which are positive, would be required to be 95% certain that the lot contains less than 20% defective units. Eq. 6.3 can equally be used for any required value of Pa and tolerable prevalence, p, to determine the number of samples required. (Using a sampling plan with c = 2, would require 30 samples to be taken to be 95% confident that the batch did not exceed 20% defectives).

With both of these schemes, however, the producer’s risk is increased as shown in Fig. 6.2 by the n = 14, c = 0 plan, while requiring fewer samples, results in greater producer’s risk.

6.6.3 Consumer Risk and Producer Risk

As discussed in the previous section, since decisions to accept or reject lots are made on samples drawn from the lots, occasions will arise when the sample results do not reflect the true condition of the lot. As discussed above producer’s risk describes the probability that an acceptable lot will be incorrectly rejected. Consumer’s risk describes the probability that an unacceptable lot when tested will be inappropriately accepted, i.e., the probability of accepting a lot whose actual microbial quality is substandard as specified in the sampling plan, even though the determined values indicate accept-able quality. The consumer’s risk is expressed by the probability of acceptance (Pa) of the batch given

6.6 The Operating Characteristic Function

138

its true level of contamination, i.e., the lower the value of Pa the less likely it is that consumers will be exposed to substandard products. Conversely, the producer’s risk for any level of contamination and sampling plan is expressed by the probability of rejection, Pr = 1 − P^a. The consumer’s risk (Pa) and producer’s risk (Pr = 1 − P^a) are depicted in Fig. 6.1.

6.6.4 Stringency and Discrimination

When sampling plans are compared and their reliability or “stringency” in making decisions is con-sidered, different aspects of their performance can be addressed. Figure 6.3 illustrates an idealized OC curve for a sampling plan that provides perfect discrimination between acceptable and unacceptable lots because acceptance probabilities drop from 100% to 0 at the chosen limit of acceptability. Any lot in which the proportion of defective units exceeds the threshold for tolerance (20% in this example) will be detected with absolute certainty, and rejected.

No practical sampling plan can achieve perfect discrimination between acceptable and unaccept-able lots because it would require all samples in the lot to be tested to identify an unacceptunaccept-able lot.

The steeper the curve, however, the closer the plan approaches that condition. Generally, steeper curves can only be achieved by increasing the number of sample units (n) to be drawn from a lot. This is illustrated in Fig. 6.4a, b which show the effect of sampling on consumer’s and producer’s risks as a function of the number of sample units tested. In the figure the limit for acceptance is 20% defective units within the lot. It can be seen that the probability of acceptance of a lot that has greater than the acceptable frequency of defective units (i.e., the consumer’s risk) declines more rapidly when n is larger. Similarly, the probability of rejection of an acceptable batch (i.e., the producer’s risk) also declines more rapidly when n is larger.

Considering the example in Sect. 6.6.2, if n = 100 and c = 20 (i.e., still tolerating 20% defective units), a batch with 15% defective units will be accepted 93% of the time while a batch with 25%

defective units will be rejected 15% of the time while a batch with 40% defective units will be rejected nearly every time (Pa < 0.00002). In summary decisions have to be made balancing the number of

Fig. 6.2 Comparison of c = 2, n = 30 (blue line) and c = 0, n = 14 (red line) sampling schemes designed to reject batches with >20% defective units with 95% confidence. The n = 14, c = 0 plan results in increased producer’s risk, but involves fewer samples

6 Concepts of Probability and Principles of Sampling

139

samples (making the OC curve more steep/moving to lower levels of defective units) and the c value (a higher value makes the curve move to the right). Together, these elements determine the steepness, location and shape of the OC curve.

Fig. 6.3 The operating characteristic (OC) curve for the idealized situation of complete discrimination between lots with a proportion of defective sample units below 20% and such lots with such a proportion above 20%

Fig. 6.4 Operating characteristic curves a) using various sampling units n = 5, n = 10, and n = 30 and b) having various combinations of n and c with c equal to 20%

of n (n = 10, c = 2; n = 25, c = 5; n = 100, c = 20)

6.6 The Operating Characteristic Function

140