Supplemental digital content 2. Quality appraisal table

(1)

Supplemental digital content 2. Quality appraisal table

Measure Name Cited by Reliability Validity Usability

Content validity

Construct validity

Criterion- related validated validity

Interpretab ility

Burden/feasib ility

Acceptabilit y

Availability

Child HCAHPS

Subscale: Attention to safety and comfort

Bruyneel et al.

(2017)(a)³⁸

IC: Not assessed

TRR: Not assessed

Yes;

-pilot test and pre-test panel where survey was reviewed by parents of children.

Yes;

evidence of confirmatory factor analysis.

Yes Not assessed Not assessed Not assessed Yes

Toomey et al.

(2015)³⁹

IC: Yes;

Spearman Brown formula for reliability

>0.7

TRR: Not assessed

Yes;

-literature review -expert panel -survey reviewed by a focus group of patents before being further revised or retested -pilot test.

Yes;

evidence of exploratory factor analysis.

Yes;

correlations of composites and single- item measures with overall hospital performance.

Not assessed Not assessed Yes;

cognitive interviews with parents to assess understanda bility.

Yes

Toomey et al.

(2016)⁴⁰

IC: Yes;

Spearman Brown formula for reliability

>0.7

TRR: Not assessed

Not assessed Not assessed Not assessed Not assessed Not assessed Not assessed Yes

(2)

Ehwerhemuep ha et al.

(2018)⁴¹

IC: Not assessed

TRR: Not assessed

Not assessed Not assessed Not assessed Yes Not assessed Not assessed Yes

Cancer Patients Experiences Questionnaire (CPEQ)

Subscale: Patient safety

Iversen et al.

(2012)⁴²

IC: No;

Cronbach’s alpha of 0.61 and 0.67 for safety subscale

TRR: No;

correlation coefficient of 0.57 for outpatients and 0.62 for inpatients

Yes;

-expert panel review -literature review to inform survey development -semi- structured interviews with patients -pilot test of survey -pre-test panel where survey was reviewed by a focus group of experts before being further revised or retested.

Yes;

evidence of EFA and CFA.

Yes; scores correlated with patient experience scores.

Yes Not assessed Not assessed Yes

Consumer Quality Index Inpatient Hospital Care (CQI IHC) (1) Subscale: Safety

Krol et al.

(2015)⁴³

IC: Yes;

Cronbach’s alpha of

>0.69 for safety subscale

(3)

TRR: Not assessed

Consumer Quality Index Inpatient Hospital Care (CQI IHC) (2)

Subscale: Feeling of safety

Smirnova et al.

(2017)⁴⁴

IC: No;

Cronbach’s alpha of 0.64 for safety subscale

TRR: Not assessed.

Not assessed Yes;

evidence of CFA.

Not assessed Yes Not assessed Not assessed Yes

The Flemish Patient Survey (FPS)

Subscale: Safe care

Bruyneel et al.

(2017)(b)⁴⁵

IC: Not assessed

TRR: Not assessed

Yes;

-literature review to inform survey development -focus groups -preliminary field test.

Yes;

evidence of both EFA and CFA.

Yes; correlated with HCAHPS scores.

Not assessed Not assessed Not assessed Yes

HEALTHQUAL Subscale: Safety

Lee et al.

(2017)⁴⁶

IC: Yes;

Cronbach’s alpha >

0.69 for safety subscale

TRR: Not assessed

Yes;

-literature review to inform survey development -expert panel review -pilot study

Yes;

evidence of CFA and EFA.

Not assessed Not assessed Not assessed Not assessed No

(4)

Press Ganey Medical Practice survey (factors revised) (MPS)

Subscale: Patient safety and health

Johnson &

Russell (2015)⁴⁷

IC: Yes;

Cronbach’s alpha

TRR: Not assessed

Not assessed Yes;

evidence of CFA

Not assessed Not assessed Assessed, but considered too lengthy.

Not assessed Yes

Johnson et al.

(2016)⁴⁸

IC: Yes;

Cronbach’s alpha

TRR: Not assessed

Nottingham et al. (2018)⁴⁹

IC: Yes;

Cronbach’s alpha

TRR: Not assessed

Not assessed Yes;

evidence of CFA

Yes; scores correlated with overall model of care and likelihood of recommending practice.

The Needs of Adults in Intensive Care Units (NAPinICU)

Subscale: Need to feel safe (needs met by staff)

Aro et al.

(2012)⁵⁰

IC:

Cronbach’s alpha

TRR: Not assessed

Yes;

-literature review to inform survey development -expert panel review -pretest panel of patients,

Not assessed Not assessed Not assessed Assessed, but considered too lengthy.

Not assessed Yes

(5)

Patient Perspective on care and Rehabilitation (PaPeR)

Subscale: Respect and safety

Wressle et al.

(2006)⁵¹

IC: No;

Cronbach’s alpha 0.68 for safety subscale

TRR: Not assessed

Yes;

-literature review -expert panel where survey was reviewed by a group of experts before being further refined or revised -qualitative interviews w/ patients -pilot study performed.

Yes;

evidence of EFA performed.

Not assessed Not assessed Assessed; time for completion up to 20 mins

Not assessed Yes

Primary Care Patient Measure of Safety (PC PMOS)

Entire questionnaire

Hernan et al.

(2016)²⁵

IC: Not assessed

TRR: Not assessed

Yes;

-expert panel review -literature review to inform survey development -pre-test panel where survey was reviewed by a focus group of experts before being further revised or retested.

Not assessed Not assessed Not assessed Not assessed Yes; formal assessments carried out to test understanda bility and readability.

Yes

(6)

Primary Care Patient Measure of Safety (PC PMOS) 28-item version

Giles et al.

(2019)⁵²

IC: No; no overall Cronbach’s alpha provided, and inadequate Cronbach’s alpha (<.69)/Inte r-item correlation (<.2 or

>.5) for 5 of 9 of safety domains

TRR: No;

correlation coefficient s not above 0.7 for 5 of 9 safety domains

No; no evidence of content validity within instrument, but reference made to previous validation before adaptation.

Yes;

evidence of CFA

Yes; assessed by correlating scores with the staff AHRQ MOS on patient safety culture (correlation not significant)

Yes Yes No No;

copyrighted

Patient

Experiences on Patient Safety (PEPS)

Sahlström et al. (2014)⁵³

IC: Yes, Cronbach’s alpha

>0.69 for all safety domains

TRR: Not assessed

Yes;

-literature review -expert panel review -pre-tested on patients.

Not assessed Not assessed Not assessed Not assessed Not assessed Yes

(7)

Sahlström et al. (2019)⁵⁴

IC: Yes, Cronbach’s alpha >

0.69 for all safety domains

TRR: Not assessed

Patient Measure of Safety (PMOS) Entire measure

Baxter et al.

(2018)⁵⁵

IR: Not assessed

TRR: Not assessed

Giles et al.

(2013)⁵⁶

IR: Not assessed

TRR: Not assessed

Yes;

-reviewed by a focus group of experts and patients before being further revised or retested.

Not assessed Not assessed Not assessed Yes;

respondents able to understand items and complete questionnaire in short timeframe, and satisfied with length.

Yes; Tested for

readability using formal assessments.

Yes

Lawton et al.

(2015)⁵⁷

IC: Yes;

Cronbach’s alpha

>0.69 for overall measure.

Individual domains not reported

Correlated with other quality and safety indicators.

Yes Assessed;

large number of items, onerous to complete, up to15 mins to completion.

Not assessed Yes

(8)

TRR: Not assessed

Lawton et al.

(2017)⁵⁸

IC: Yes;

Cronbach’s alpha

>0.69 for overall measure.

Individual domains not reported

TRR: Not assessed

McEachan et al. (2014)⁵⁹

IC: Yes, Cronbach’s alpha

>0.69 for 5 of 8 safety domains

TRR: No;

Pearson’s r of .75 for PMOS positive index, but

>.7 for only 2 of 8 safety domains

Not assessed Yes;

evidence of exploratory factor analysis.

Yes;

correlations with staff questionnaire scores.

(9)

O’Hara et al.

(2016)⁶⁰

IC: Not assessed

TRR: Not assessed

PMOS-30

Louch et al.

(2019)⁶¹

Yes;

>.69 for overall measure, and inter- item correlation s within acceptable ranges for all safety domains

TRR: Not assessed

Yes;

‘shortening exercise’

included an iterative consensus approach with researchers, and pre-test patient panel.

Not assessed Not assessed Yes Yes;

completion could be facilitated by a research nurse or volunteer.

Yes; ‘easy read’

versions have been produced.

Yes

PMOS-10

Louch et al.

(2019)⁶¹

Yes;

>.69 for overall measure, individual domains not reported

TRR: Not assessed

Yes; pre-test patient panel.

Not assessed Yes; correlated with measure of patient experience.

Yes Yes; quick to complete.

Completion could be facilitated by a research nurse or volunteer,

Yes; ‘easy read’

versions have been produced.

Yes

(10)

PMOS-A

Taylor et al.

(2016)⁶²

IC: Yes;

>0.69, or inter-item correlation of >.2 for 8 of 9 safety domains

TRR: Not assessed

Yes;

-pre-test panel where survey was reviewed by a focus group of patients.

Not assessed Not assessed Yes Assessed;

areas for improvement to the type and format of resources that need to be provided for data collection.

Not assessed Yes

Taylor et al.

(2019)⁶³

IC: No;

>0.69 for 4 of 9 subscales

TRR: Not assessed

No;

evidence of content validity within instrument, but reference made to previous validation.

Yes;

evidence of confirmatory factor analysis

Yes; PMOS-A correlated with adherence to clinical guidelines.

Patient Reported Experiences and Outcomes of Safety in Primary Care (PREOS- PC)

Entire Questionnaire

Ricci-Cabello et al. (2016)⁶⁴

IC: Yes;

Cronbach’s alpha of >

0.69 for all safety domains

TRR: No;

correlation coefficient greater than 0.7 for only 2 of 11 domains

Yes;

-literature review to inform survey development -expert panel review -pilot test -pre-test panel where survey was reviewed by a focus groups of experts

Yes;

evidence of CFA.

Yes Not assessed Not assessed Not assessed Yes

(11)

before being further revised or refined.

Ricci-Cabello et al. (2017)⁶⁵

IC: Not assessed

TRR: Not assessed

TRIQ

Entire Questionnaire

van Melle et al. (2019)⁶⁶

IC:

Guttman’s lower bound and composite reliability;

acceptable to good.

TRR: Not assessed

Yes;

-literature review -expert panel review -pre-test panel where survey was reviewed by a focus group of patients before being further revised.

Evidence of EFA.

Scores correlated to rating of satisfaction.

Not assessed Not assessed Assessed; 15- 20 mins completion.

Not assessed Yes.

Untitled Measures Subscale:

Attitudes about patient safety

An et al.

(2017)⁶⁷

IC: Yes;

Cronbach’s alpha

TRR: Not assessed

Yes;

-literature review -pre-test panel of experts -pilot test.

Construct validity tests:

Goodness of fit index;

Non-normed fit index;

Comparative fit index;

Standardized root mean square residual;

(12)

Root mean square error of

approximati on.

Subscale: Sense of safety and

receiving support

Bandurska et al. (2016)⁶⁸

IC: Not assessed

TRR: Not assessed

Not assessed Not assessed Not assessed Not assessed Not assessed Not assessed No

Subscale: Safe care Bruyneel et al.

(2018)⁶⁹

IC: Not assessed

TRR: Not assessed

Yes;

-literature review -patient and stakeholder focus group to inform development -pre-test panel.

Yes;

evidence of EFA and CFA.

Yes; criterion- related validity.

Multilevel logistic regression analysis used to estimate the association between factors and global rating of the hospital.

Not assessed Addressed;

input was too time‐

consuming, short time span in which the data collection and uploading needed to be completed.

Not assessed Yes

Cox et al.

(2013)⁷⁰

IC: Not assessed

TRR: Not assessed

Yes;

-literature review -expert panel and patient review -pilot test of survey.

Yes;

evidence of CFA.

Subscale:

Perceptions of safety

Desmedt et al.

(2017)¹⁹

IC: Not assessed

TRR: Not assessed

Not assessed Not assessed Yes; safety items

correlated with PACIC scores.

(13)

Dixon et al.

(2015)⁷¹

IC: Not assessed

TRR: Not assessed

Yes; - literature review -expert panel review -pre-test panel where survey was reviewed by a group of experts.

Not assessed Not assessed Not assessed Not assessed Not assessed Yes

Subscale: Safety indicators

Duggirala et al.

(2008)⁷²

IC: No;

TRR: Not assessed

Yes;

-literature review -pilot study of survey where survey was reviewed by respondents before being further revised.

Evidence of CFA

Yes;

scale scores correlated with overall patient satisfaction

Subscale:

Quality/safety

Malik et al.

(2016)⁷³

IC:

Cronbach’s alpha

TRR: Not assessed

Yes;

-literature review -pre-test panel where survey was reviewed by experts before being revised.

Evidence of EFA;

evidence of inter- dimensional correlation assessment (scores correlated with overall satisfaction).

Not assessed Yes Not assessed Not assessed Yes; in table.

Subscale: Feeling safe

Robinson et al.

(2018)⁷⁴

IC:

Cronbach’s alpha

Yes;

-literature review

Not assessed Not assessed Not assessed Not assessed Not assessed Yes; in appendix.

(14)

TRR: Not assessed

-focus groups

Subscale:

confidence in the quality and safety

Schmidt et al.

(2013)⁷⁵

IC: Not assessed

TRR: Not assessed

Yes;

-literature review

Not assessed Yes; scores correlated with clinic PCMH score

Yes Not assessed Not assessed Yes; in table.

Subscale:

Perceptions of safety and staff technical skills

Schwappach et al. (2003)⁷⁶

IC: Not assessed

TRR: Not assessed

Not assessed Not assessed Not assessed Not assessed Not assessed Not assessed Yes; in table.

Scott et al.

(2016)⁷⁷

IC: Not assessed

TRR: Not assessed

Yes;

-focus groups carried out to develop survey -pre-test panel where survey was reviewed by a focus group before being further revised or retested.

Not assessed Not assessed Not assessed Not assessed Not assessed Yes; in table.

Scott et al.

(2019)⁷⁸

IC: Not assessed

TRR: Not assessed

Not assessed Not assessed Not assessed Not assessed Addressed;

staff members interviewed and suggested that time constraints impacted the

Yes;

evaluation forms filled in by patient participants to examine

Yes; in appendix.

(15)

distribution to patients. Also possibility of patients being overburdened with

paperwork.

survey acceptability

Subscale: Safety measures

Padma et al.

(2010)⁷⁹

IC: No;

TRR: Not assessed

Yes;

-literature review -pilot study where survey was reviewed by a panel of experts before being further revised.

Evidence of CFA

Yes; scale scores

correlated with overall patient satisfaction

readability, language, and scale assessed by patient panel

Yes; in appendix.