3 CHAPTER III: METHODS
3.12 STATISTICAL PROCESSING AND ANALYSIS .1 Proficiency testing
a) Results were tabulated (Table 3) and analysed. Microsoft EXCEL (Microsoft Excel, Palisade Corp, Newfield, NY, USA) was used to quantify the variables listed in chapter 3.8.
b) Significance of differences between variables (proportions) was determined by
•2-n
calculating p-values using the Mantel-Haenzel Chi squared test (X Test).
3.12.2 Classification of errors
Proficiency testing results were classified as indicated in table 3 below.
Table 3: Cross classification and tabulation of proficiency testing errors Expected
Negative Scanty 1+
2+
3+
result
Negative Correct LFP HFP HFP HFP
Result of microscopy centre Scanty
LFN Correct QE QE QE
1+
HFN QE Correct QE QE
2+
HFN QE QE Correct QE
3+
HFN QE QE QE Correct LFP = Low false positive; HFP = High false positive; HFN = High false negative;
QE = Quantification error; LFN = Low false negative
3.12.3 Scoring system
Laboratories reporting correct results were awarded a maximum possible score of 10 points per slide. Incorrect results i.e. high false positive and high false negative were awarded a score of zero. Low false positive and low false negative slides had five points deducted, giving these slides a score of five points. Slides with quantification errors were awarded a score often points.
Scoring was done as indicated in table 4 below.
Table 4: Point allocation according to classification of error Error
Correct QE LFN
LFP HFN HFP
Points 10 10 5 5 0 0
3.12.4 Criteria for assessment for performance
Criteria for assessment of performance in this study was adapted from recommendations by the American Public Health Laboratory and Centre for Disease Control and from the study conducted in Limpopo, South Africa. [11, 13]
• Errors: The target for optimal performance was for laboratories not to have any errors of any type.
• Major Errors: Any major error (HFP or HFN) would indicate unacceptable performance and should trigger an evaluation and corrective action if needed. It is
possible that no significant problems in laboratory practice would be identified, and performance trends should be monitored over time.
• Minor errors: Minor errors (LFP or LFN) would require further evaluation if they exceed the average number seen in all TB microscopy centres in the province or if the number of minor errors over time demonstrates a trend.
• Quantification errors: Disagreement on bacillary concentration is less serious and is not usually calculated as a percentage error. Quantification errors are not that important, as they do not drastically influence the decision making on patient management. This type of error only distinguishes the good from the very good microscopist.
Overall Disagreement on false positivity and false negativity should be less than 5%.
3.12.5 Assessment of overall performance of the province
To calculate overall performance for the province, all the scores for all the laboratories were combined and divided by the number of participating laboratories and calculated as percentage correctness in microscopy diagnosis for the province. The overall aim was to reach 95% agreement, between all the participating laboratories and the reference standard, for the province as a whole.
3.12.6 Assessment of overall performance of individual laboratories
To calculate overall performance for individual laboratories, all the scores for all the slides processed by each laboratory were combined and calculated as percentage
correctness in microscopy diagnosis for the laboratory. The overall aim was be to reach 95% agreement (acceptable performance), in reading proficiency testing slides, between the various laboratories and the reference standard.
It was recommended that should the overall disagreement (error) exceed the accepted critical value of 5% (proportion discordant among negative and positive slides, or overall discordance), then the entire procedure, including quality of smear preparation, reagents,
staining method and reading should be reviewed by the supervisor and the technologist/technician concerned. [13]
3.12.7 Calculation of sensitivity, specificity, positive predictive value and negative predictive value
Results were analysed on 2 x 2 tables as indicated in Table 5 below.
Table 5: Analysis of Results on 2 x2 Tables
Reference Material
Positive Negative Total
Laboratory Positive a b (a+b) Result Negative c d (c+d)
Total (a+c) (b+d) a+b+c+d
The formulas for calculating sensitivity , specificity , positive predictive value and negative predictive26 value are as follows:
• Sensitivity = [a/(a+c)] x 100
• Specificity = [d/(b+d)] x 100
• Positive Predictive Value = [a/(a+b) x 100]
• Negative Predictive Value = [d/(c+d) x 100]
Overall sensitivity, specificity, positive predictive value and negative predictive value for the province were calculated using the reference laboratory as a comparator.
The targets for sensitivity, specificity, positive predictive value and negative predictive value were as follows:
23 Percentage of positive test result out of all true positives
24 Percentage of negative test results out of all true negatives
25 Percentage of positive test result that are truly positive
26 Percentage of negative test results which are truly negative
• Sensitivity: Since the low positive slides were removed from the study all other positive slides should be read correctly. However, 100% sensitivity is very difficult to achieve even by the controller, therefore sensitivity was set at 95%. [11]
• Specificity: Any false negative results would indicate unacceptable performance and should trigger corrective action. However, Minor errors, do occur occasionally even in laboratories that are performing well. Therefore the target for specificity was set at 95%. [11]
• Positive predictive value: Any false positive results would indicate unacceptable performance and should trigger corrective action. However, Minor errors, do occur occasionally even in laboratories that are performing well, therefore the target for positive predictive value was set at 95%
• Negative predictive value: Any false negative results would indicate unacceptable performance; However, Minor errors, do occur occasionally even in laboratories that are performing well. Therefore the target for negative predictive value was set at 95%.
The above recommendations were adopted to assess the proficiency of TB smear microscopy in public health TB microscopy laboratories in KwaZulu-Natal.
A comparison was made with the National Health Laboratory Service proficiency testing results, as they were conducting proficiency testing for the year 2006 and would therefore provide additional knowledge to the status of TB microscopy services in KwaZulu-Natal.
Analysis of the National Health Laboratory Service data would also form the basis for comparison with the other provinces in the country.
3.12.8 Key informant interviews
Each issue discussed in the interview was categorized, grouped and summarized.