Supplemental Digital Content Manuscript
Gero, Muller et al. A Standardized Methodology for Benchmarking in Surgery based on an International Expert Delphi Consensus. Annals of Surgery. 2020
Table of contents
Supplementary Figures
Supplementary Figure 1. Global benchmark cut-offs allow unbiased comparison of outcomes across centers.
Supplementary Tables
Supplementary Table 1. 1
stround of the Delphi questionnaire.
Supplementary Table 2. The panel of experts.
Supplementary Table 3. Consensual recommendations of the Delphi expert panel on the identification and implementation of outcome benchmarks in surgery.
Supplementary Table 4. Items of the Delphi study that failed to reach consensus.
Figures
Supplementary Figure 1. Global benchmark cut-offs allow unbiased comparison of outcomes
across centers. By comparing the outcomes of benchmark patients only, differences in
institutional case-mix do not cofound comparability. Benchmark cases refer to a low-risk
patient cohort that underwent a standardized intervention. CCI: Comprehensive complication
index
19Supplementary Table 1. 1
stround of the Delphi questionnaire.
Question
number How to define low-risk patients?
1
Benchmark criteria should exclude patients with an expected higher morbidity. Which criteria should be applied?
►Documented risk factors associated with inferior post-surgical outcomes (e.g.: ischemic heart disease, renal impairment, diabetes, elderly age, BMI, etc.)
►Specific medications that are associated with inferior post-surgical outcomes (e.g.: immunosuppression and anticoagulation)
►Score based cut-offs: ASA, ACS NSQIP Surgical Risk Calculator (https://riskcalculator.facs.org/RiskCalculator/), P-POSSUM score (http://www.riskprediction.org.uk/index-pp.php), etc..
2 Should benchmark criteria exclude patients with any type of previous surgery in the same body compartment (e.g.: no previous abdominal surgery for gastrectomy benchmark patients)?
Do you have any other suggestions on criteria to identify benchmark patients?
How to select interventions to be benchmarked?
3 Should benchmark criteria exclude patients with any associated procedure during the index procedure (e.g.:
cholecystectomy during gastric bypass)
4 Criteria to identify benchmark interventions should differentiate between malignant and benign indications (e.g.: pancreaticoduodenectomy)
Do you have any other suggestions on criteria to identify benchmark interventions?
How to define participating centers / data collection strategies?
5
Criteria to select centers for a benchmark study should include "hospital volume"?
If Yes, which cutoff would you suggest for your specialty (e.g.: 20 cases/year)?
6
Criteria to select centers for a benchmark study should include: "surgeon-volume"?
If Yes, which cutoff would you suggest for your specialty (e.g.: 20 cases/year)?
7
Criteria to select centers for a benchmark study should require the inclusion of the following level of care centers:
►Primary
►Secondary
►Tertiary referral
►Academic
►All of the above
8 Participating centers should have at least one previous peer-reviewed publication on surgical outcomes of the procedure which is to be benchmarked?
Do you have any other suggestions on center selection?
How and how often should benchmark values computed?
9 What should be the study period (e.g.: all consecutive liver transplantations performed on benchmark patients during a 5-year period)?
10 What should be the minimum required follow-up period for all benchmark patients (e.g.: all benchmark patients should have a documented follow-up of 6-months)?
11 How often should the benchmark values be updated (duration of validity of the results in years)?
12
The benchmark cut-off has been defined at the 75th percentile of the median value for each outcome indicator (CCI, Survival, Reoperation, etc.) of the studied benchmark patients. (Ref.: https://goo.gl/gWXUop). Do you agree with this method or would suggest another cut-off?
Do you have any other suggestions on study design?
How to use benchmarks in clinical practice?
13
Which of the following items is/are necessary to improve accuracy of outcome reporting?
►Physician maintained surgical database
►Study nurse / data-manager maintained surgical database
►Externally audited surgical database
►Central adjudication committee to select benchmark cases
►Prospective observational study design (NOT retrospective analysis of prospective databases)
14
What measure would you take to close the performance gap between your center and global benchmarks (best achievable outcomes)?
►Detect surgeons who need additional training
►Discuss results with the entire department, including the para-medical staff
►Discuss potential strategies for improvement with the hospital management
►Systematically present patients below the benchmarks at morbidity-mortality conferences
►Change the surgical portfolio of your center
15 Should regulatory bodies such as insurance companies or governmental agencies perform benchmarking rather than the surgical community?
16 Would you recommend to make each hospital’s outcomes publicly available, to allow comparison with the global benchmark of the respective surgical procedures?
17 Should benchmarks be used to compare outcomes of a given surgical intervention with a completely different procedure (e.g.: esophagectomy vs. liver resection)?
18 Would you recommend including a specific threshold for operating a certain amount of benchmark and non- benchmark cases in the curricula of surgical trainees?
19
Would you recommend using “benchmark” patient status to assist the preoperative decision on the
composition of the surgical team? (e.g.: benchmark cases to be taught to residents, non-benchmark cases to be performed by senior surgeon)
20 Would you recommend using “benchmark” patient status to assist the informed consent process with the patients?
Do you have any other comments on benchmarking in surgery?
Supplementary Table 2. The panel of experts.
Specialty Expert Affiliation Continent
Bariatric
Rajesh Aggarwal Thomas Jefferson Univ. Hospitals, Philadelphia, PA, USA Northern America Kelvin Higa University of California San Francisco, Fresno, CA, USA Northern America Jacques Himpens St Pierre University Hospital, Brussels, Belgium Europe
Thorsten Olbers Linköping University, Norrköping, Sweden Europe
Francois Pattou University Hospital Lille, Lille, France Europe
Ralph Peterli University of Basel at Claraspital Basel, Basel, Switzerland Europe Matias Sepulveda Hospital Dipreca, Santiago de Chile, Chile Southern America
Torgeir T. Søvik Oslo University Hospital Ullevaal, Norway Europe
Michel Suter Riviera-Chablais Hosp./University Hospital Lausanne, Switzerland Europe
Breast
Sally E Carty University of Pittsburgh, Pittsburgh, PA, USA Northern AmericaCheng-har Yip University of Malaysia, Kuala Lumpur, Malaysia Asia
Colorectal
David H. Berger Baylor St. Luke's Medical Center, Houston, TX, USA Northern America Sebastiano Biondo Bellvitge University Hospital, University of Barcelona, Spain Europe
Robin McLeod University of Toronto, Toronto, ON, Canada Northern America Ronan O’Connell St Vincent’s University Hospital, Dublin, Ireland Europe
Mathias Turina University Hospital Zurich, Zurich, Switzerland Europe
Desmond Winter University College Dublin, Dublin, Ireland Europe
Endocrine
Sonia Sugg University of Iowa Health Care, Iowa City, IA, USA Northern America Frederic Triponez University Hospital Geneva, Geneva, Switzerland Europe
Sandra Wong Dartmouth Geisel School of Medicine, Hanover, NH, USA Northern America
Esophageal
Luigi Bonavina University of Milan, Milano, Italy Europe
Misha Luyer Catharina Hospital Eindhoven, The Netherlands Europe
Magnus Nilsson Karolinska University Hospital, Stockholm, Sweden Europe Francesco Palazzo Thomas Jefferson Univ. Hospitals, Philadelphia, PA, USA Northern America
Wolfgang Schröder University of Cologne, Cologne, Germany Europe
David Watson Flinders University, Adelaide, Australia Australia
Hepato- biliary, pancreas
Marc Besselink University of Amsterdam, The Netherlands Europe
Kevin Conlon The University of Dublin, Dublin, Ireland Europe
Keith Lillemoe Johns Hopkins University, Baltimore, MD, USA Northern America
Marek Krawczyk Medical University of Warsaw, Poland Europe
Peter A Lodge St James's University Hospital, Leeds, UK Europe
Martin de Santibanes Hospital Italiano de Buenos Aires, Argentinia Southern America
Liver transplant
Bo-Göran Ericzon Karolinska Institute, Stockholm, Sweden Europe
Pål-Dag Line Oslo University Hospital, Norway Europe
Paolo Muiesan University of Birmingham, UK Europe
Wojciech G Polak Erasmus University Hospital, Rotterdam, The Netherlands Europe
Thoracic
Shanda Blackmon Mayo Clinic, Rochester, MN, USA Northern America
Gilbert Massard University Hospital Strasbourg, France Europe
Enrico Ruffini University of Torino, Torino, Italy Europe
Vascular / cardiac
Mario Lachat University Hospital Zurich, Zurich, Switzerland Europe
Anonymous Expert Undisclosed Undisclosed
Supplementary Table 3. Consensual recommendations of the Delphi expert panel on the identification and implementation of outcome benchmarks in surgery.
Agreement (%)
New feature / Validation of current practice How to select interventions to be benchmarked?
Emergency and elective procedures should be benchmarked separately 100 Current practice Operations performed for malignant and benign diseases should be benchmarked separately 92.5 New feature Additional procedures should not be associated to the index procedure
(i.e.: cholecystectomy during gastric bypass) 70 Current practice
How to define low-risk patients?
Stage of disease (i.e. TNM stage; obesity stage for bariatric benchmarks, etc.) 91 Current practice Documented pre-operative risk factors (i.e. diabetes, elderly age, BMI, etc.) 88 Current practice Pre-operative medications affecting outcomes (i.e. immunosuppression and anticoagulation) 85 Current practice Score based cut-offs: ASA, ACS-NSQIP Surgical Risk Calculator, P-POSSUM score, etc. 80 Current practice Previous surgery in the same body compartment is not an exclusion criterion 70 New feature How to define participating centers / data collection strategies?
Criteria should include "hospital volume" 95 Current practice
Presence of a certified multidisciplinary team
(intensive care unit, interventional radiology, medical specialties, etc.) 94 New feature Participation in official national and/or international outcome registries 91 New feature Presence of minimum 2 board-certified surgeons in the benchmarked specialty 88 New feature Ability to offer a variety of procedures and revision operations
(i.e.: for bariatric surgery at least 3 different) 85 New feature
Criteria should include "surgeon-volume" 80 New feature
Participation in accredited “quality of excellence” program (or equivalent schemes) 70 New feature How and how often should benchmark values be computed?
The benchmark cut-off is defined at the 75th percentile of the median value for each outcome
indicator 95 Current practice
Update the benchmarks every 3 years, or earlier if new treatment modalities are introduced 88 New feature Joint-venture between surgeons and government agencies, without the involvement of
insurance companies 88 New feature
Minimal study period: 4 years 79 Current practice
Externally audited surgical databases are mandatory 70 New feature
How to use benchmarks in clinical practice?
Discuss off-benchmark outcomes within the department, including para-medical staff and
hospital management 85 New feature
Benchmarks should not be used to compare outcomes of a given surgical intervention with a
completely different procedure (i.e.: esophagectomy vs. liver resection) 80 New feature Discuss cases with off-benchmark outcomes systematically at morbidity-mortality conferences 80 Current practice
“Benchmark” patient status could be used to assist the preoperative decision on the
composition of the surgical team (i.e.: benchmark cases selected for teaching) 79 New feature Identify surgeons in need for additional training to close the eventual performance gaps * 79 New feature Make hospitals' outcomes publicly available to allow comparison with the global benchmarks 73 New feature