NOTICE: This document contains correspondence generated during peer review and subsequent revisions but before transmittal to production for composition and copyediting:
• Comments from the reviewers and editors (email to author requesting revisions)
• Response from the author (cover letter submitted with revised manuscript)*
*The corresponding author has opted to make this information publicly available.
Personal or nonessential information may be redacted at the editor’s discretion.
Questions about these materials may be directed to the Obstetrics & Gynecology editorial office:
Date: Jan 29, 2021
To: "Evan Robert Tannenbaum"
From: "The Green Journal" [email protected] Subject: Your Submission ONG-20-3282
RE: Manuscript Number ONG-20-3282
Reputational risk: a randomized trial of how a resident’s reputation impacts laparoscopic skills assessment Dear Dr. Tannenbaum:
Your manuscript has been reviewed by the Editorial Board and by special expert referees. Although it is judged not acceptable for publication in Obstetrics & Gynecology in its present form, we would be willing to give further consideration to a revised version.
If you wish to consider revising your manuscript, you will first need to study carefully the enclosed reports submitted by the referees and editors. Each point raised requires a response, by either revising your manuscript or making a clear and convincing argument as to why no revision is needed. To facilitate our review, we prefer that the cover letter include the comments made by the reviewers and the editor followed by your response. The revised manuscript should indicate the position of all changes made. We suggest that you use the "track changes" feature in your word processing software to do so (rather than strikethrough or underline formatting).
Your paper will be maintained in active status for 21 days from the date of this letter. If we have not heard from you by Feb 19, 2021, we will assume you wish to withdraw the manuscript from further consideration.
REVIEWER COMMENTS:
Reviewer #1:
Well written and researched paper on evaluation of a trainee surgical skills. The introduction of bias in evaluation is important and readers of the journal will be aided by this research. Use of video evaluation of procedures may mitigate bias as you point out.
Questions: Is the Likert scale appropriate?
There seems to be overlap in SD in all groups, can you address?
Could you expand on other surgical specialties that have done similar research?
Reviewer #2:
Precis - faxulty assessment of residents is influenced by assessor's knowledge of past performance Abstract - Objective - quantify effect of resident's reputation on assessment skills
Methods - faculty randomized to 1 of 3 hypothetical resident scenarios - resident with high, average, or low surgical skills view video of same resident performing l/s salpingoooporectomy and assessment with OSATS
Results - assessment scores on OSATS (out of 20) and on global score (out of 5) varied by reputation:
high resident - 14.8/3.7, average 13.5/3.4, low - 11.2/2.9
Conclusion - assessment is influenced by knowledge of past performance and knowledge introduces bias that affects scores Intro - work place- based assessments used but are subject to bias - unreliable and unfair data; to mitigate bias - measure bias - cultural and systems changes to mitigate bias
video based assessments to quantify effect of reputation as form of bias
Methods - multi-center, single blind, random study - high, average or remedial group, conducted at 9 sites with video of surgery
stated purpose to assessors was to explore the reliability of OSATS so they wouldn't be biased in assessment ; 8 faculty needed per group
Results - 43 observers - 14.8/13.5/11.3 and 3.7/3.4/2.9 for the 2 scores
View Letter ...
1 of 6 2/16/2021, 12:46 PM
Discussion - bias exists in workplace based assessments and reputation affects assessments
need fair and accurate assessment data to detect underperformance of those with good reputation and to allow appropriate evaluation of those with remedial reputation
data is needed to support the validity of video based assessments but reputation affects assessment so objective assessment is needed
Comments -
This is an interesting study on the impact of bias - the findings are concerning, though not at all surprising.
I imagine the same findings could be found with residents who are well liked versus not as well liked and this would be a tool to assess that.
I would include some data as to what is considered to be a significant variation for the OSATS or the global scale - though these numbers are statistically significant, what constitutes a meaningful difference in assessment?
Reviewer #3:
This manuscript explores a type of unconscious bias in the evaluation of learners in Ob-Gyn residency. Namely, the impact of resident reputation on assessment of laparoscopic skills using what was designed to be an "objective" evaluation (OSATS). Through a simple, but elegant design, the investigators highlight a potential source of bias even using the OSAT tool that by design is meant to be a standardized approach to resident assessment. The idea of implicit bias is not new, but the results of this study highlight a critical area of concern in the evaluation of learners even when using objective and standardized tools.
STATISTICS EDITOR COMMENTS:
Lines 139-141: How many observers rated the videos and was the 3 of 5 score an average, or a consistent grade among all observers? Need to clarify re: the video used for the rest of the study.
lines 166-172: The sample size calculation needs clarification. It appears that the Authors were stipulating a 2 point difference in means, with a SD = 2, equal cohort sizes, power = 80% and alpha = .05. In order to compare two groups, the required sample size = 17 in each group. The ANOVA test would test whether the overall data distribution did/did not conform to no difference, but it would not test specific pairs. It appears that the Authors were positing that the high, average and low performing groups would differ by increments of 2 in their mean values, which is not what was tested by ANOVA. Furthermore, the sample size does not allow simultaneous testing of the 5 scale Likert comparison.
The 5 point Likert scale should have been formatted as median(range or IQR) and tested non-parametrically and presented as secondary outcome. Given the sample sizes, the OSATS should also be statistically evaluated using a non-parametric test (Kruskal-Wallis) for the overall test with pair wise testing or using one group as the referent for testing between groups.
Fig 1: See comments on use of means to summarize the groups.
EDITORIAL OFFICE COMMENTS:
1. The Editors of Obstetrics & Gynecology are seeking to increase transparency around its peer-review process, in line with efforts to do so in international biomedical peer review publishing. If your article is accepted, we will be posting this revision letter as supplemental digital content to the published article online. Additionally, unless you choose to opt out, we will also be including your point-by-point response to the revision letter. If you opt out of including your response, only the revision letter will be posted. Please reply to this letter with one of two responses:
View Letter .
2 of 6 2/16/2021, 12:46 PM
A. OPT-IN: Yes, please publish my point-by-point response letter.
B. OPT-OUT: No, please do not publish my point-by-point response letter.
2. Obstetrics & Gynecology uses an "electronic Copyright Transfer Agreement" (eCTA). When you are ready to revise your manuscript, you will be prompted in Editorial Manager (EM) to click on "Revise Submission." Doing so will launch the resubmission process, and you will be walked through the various questions that comprise the eCTA. Each of your coauthors will receive an email from the system requesting that they review and electronically sign the eCTA.
Please check with your coauthors to confirm that the disclosures listed in their eCTA forms are correctly disclosed on the manuscript's title page.
3. Clinical trials submitted to the journal as of July 1, 2018, must include a data sharing statement. The statement should indicate 1) whether individual deidentified participant data (including data dictionaries) will be shared; 2) what data in particular will be shared; 3) whether additional, related documents will be available (eg, study protocol, statistical analysis plan, etc.); 4) when the data will become available and for how long; and 5) by what access criteria data will be shared (including with whom, for what types of analyses, and by what mechanism). Responses to the five bullet points should be provided in a box at the end of the article (after the References section).
4. Responsible reporting of research studies, which includes a complete, transparent, accurate and timely account of what was done and what was found during a research study, is an integral part of good research and publication practice and not an optional extra. Obstetrics & Gynecology supports initiatives aimed at improving the reporting of health research, and we ask authors to follow specific guidelines for reporting randomized controlled trials (ie, CONSORT). Include the appropriate checklist for your manuscript type upon submission. Please write or insert the page numbers where each item appears in the margin of the checklist. Further information and links to the checklists are available at
http://ong.editorialmanager.com. In your cover letter, be sure to indicate that you have followed the CONSORT, MOOSE, PRISMA, PRISMA for harms, STARD, STROBE, RECORD, CHEERS, SQUIRE 2.0, or CHERRIES guidelines, as appropriate.
5. Standard obstetric and gynecology data definitions have been developed through the reVITALize initiative, which was convened by the American College of Obstetricians and Gynecologists and the members of the Women's Health Registry Alliance. Obstetrics & Gynecology has adopted the use of the reVITALize definitions. Please access the obstetric data definitions at https://www.acog.org/practice-management/health-it-and-clinical-informatics/revitalize-obstetrics-data- definitions and the gynecology data definitions at https://www.acog.org/practice-management/health-it-and-clinical- informatics/revitalize-gynecology-data-definitions. If use of the reVITALize definitions is problematic, please discuss this in your point-by-point response to this letter.
6. Because of space limitations, it is important that your revised manuscript adhere to the following length restrictions by manuscript type: Original Research reports should not exceed 22 typed, double-spaced pages (5,500 words). Stated page limits include all numbered pages in a manuscript (i.e., title page, précis, abstract, text, references, tables, boxes, figure legends, and print appendixes) but exclude references.
7. Specific rules govern the use of acknowledgments in the journal. Please note the following guidelines:
* All financial support of the study must be acknowledged.
* Any and all manuscript preparation assistance, including but not limited to topic development, data collection, analysis, writing, or editorial assistance, must be disclosed in the acknowledgments. Such acknowledgments must identify the entities that provided and paid for this assistance, whether directly or indirectly.
* All persons who contributed to the work reported in the manuscript, but not sufficiently to be authors, must be
acknowledged. Written permission must be obtained from all individuals named in the acknowledgments, as readers may infer their endorsement of the data and conclusions. Please note that your response in the journal's electronic author form View Letter
3 of 6 2/16/2021, 12:46 PM
verifies that permission has been obtained from all named persons.
* If all or part of the paper was presented at the Annual Clinical and Scientific Meeting of the American College of Obstetricians and Gynecologists or at any other organizational meeting, that presentation should be noted (include the exact dates and location of the meeting).
8. The most common deficiency in revised manuscripts involves the abstract. Be sure there are no inconsistencies between the Abstract and the manuscript, and that the Abstract has a clear conclusion statement based on the results found in the paper. Make sure that the abstract does not contain information that does not appear in the body text. If you submit a revision, please check the abstract carefully.
In addition, the abstract length should follow journal guidelines. The word limit for Original Research articles is 300 words;
Reviews is 300 words; Case Reports is 125 words; Current Commentary articles is 250 words; Executive Summaries, Consensus Statements, and Guidelines are 250 words; Clinical Practice and Quality is 300 words; Procedures and Instruments is 200 words. Please provide a word count.
9. Abstracts for all randomized, controlled trials should be structured according to the journal's standard format. The Methods section should include the primary outcome and sample size justification. The Results section should begin with the dates of enrollment to the study, a description of demographics, and the primary outcome analysis. Please review the sample abstract that is located online here: http://edmgr.ovid.com/ong/accounts/sampleabstract_RCT.pdf. Please edit your abstract as needed.
10. Only standard abbreviations and acronyms are allowed. A selected list is available online at http://edmgr.ovid.com /ong/accounts/abbreviations.pdf. Abbreviations and acronyms cannot be used in the title or précis. Abbreviations and acronyms must be spelled out the first time they are used in the abstract and again in the body of the manuscript.
11. The journal does not use the virgule symbol (/) in sentences with words. Please rephrase your text to avoid using
"and/or," or similar constructions throughout the text. You may retain this symbol if you are using it to express data or a measurement.
12. In your Abstract, manuscript Results sections, and tables, the preferred citation should be in terms of an effect size, such as odds ratio or relative risk or the mean difference of a variable between two groups, expressed with appropriate confidence intervals. When such syntax is used, the P value has only secondary importance and often can be omitted or noted as footnotes in a Table format. Putting the results in the form of an effect size makes the result of the statistical test more clinically relevant and gives better context than citing P values alone.
If appropriate, please include number needed to treat for benefits (NNTb) or harm (NNTh). When comparing two procedures, please express the outcome of the comparison in U.S. dollar amounts.
Please standardize the presentation of your data throughout the manuscript submission. For P values, do not exceed three decimal places (for example, "P = .001"). For percentages, do not exceed one decimal place (for example, 11.1%").
13. Please review the journal's Table Checklist to make sure that your tables conform to journal style. The Table Checklist is available online here: http://edmgr.ovid.com/ong/accounts/table_checklist.pdf.
14. Please review examples of our current reference style at http://ong.editorialmanager.com (click on the Home button in the Menu bar and then "Reference Formatting Instructions" document under "Files and Resources). Include the digital object identifier (DOI) with any journal article references and an accessed date with website references. Unpublished data, in-press items, personal communications, letters to the editor, theses, package inserts, submissions, meeting
presentations, and abstracts may be included in the text but not in the reference list.
In addition, the American College of Obstetricians and Gynecologists' (ACOG) documents are frequently updated. These View Letter
4 of 6 2/16/2021, 12:46 PM
documents may be withdrawn and replaced with newer, revised versions. If you cite ACOG documents in your manuscript, be sure the reference you are citing is still current and available. If the reference you are citing has been updated (ie, replaced by a newer version), please ensure that the new version supports whatever statement you are making in your manuscript and then update your reference list accordingly (exceptions could include manuscripts that address items of historical interest). If the reference you are citing has been withdrawn with no clear replacement, please contact the editorial office for assistance ([email protected]). In most cases, if an ACOG document has been withdrawn, it should not be referenced in your manuscript (exceptions could include manuscripts that address items of historical
interest). All ACOG documents (eg, Committee Opinions and Practice Bulletins) may be found at the Clinical Guidance page at https://www.acog.org/clinical (click on "Clinical Guidance" at the top).
15. Figures 1-2: okay
When you submit your revision, art saved in a digital format should accompany it. If your figure was created in Microsoft Word, Microsoft Excel, or Microsoft PowerPoint formats, please submit your original source file. Image files should not be copied and pasted into Microsoft Word or Microsoft PowerPoint.
When you submit your revision, art saved in a digital format should accompany it. Please upload each figure as a separate file to Editorial Manager (do not embed the figure in your manuscript file).
If the figures were created using a statistical program (eg, STATA, SPSS, SAS), please submit PDF or EPS files generated directly from the statistical program.
Figures should be saved as high-resolution TIFF files. The minimum requirements for resolution are 300 dpi for color or black and white photographs, and 600 dpi for images containing a photograph with text labeling or thin lines.
Art that is low resolution, digitized, adapted from slides, or downloaded from the Internet may not reproduce.
16. Authors whose manuscripts have been accepted for publication have the option to pay an article processing charge and publish open access. With this choice, articles are made freely available online immediately upon publication. An
information sheet is available at http://links.lww.com/LWW-ES/A48. The cost for publishing an article as open access can be found at https://wkauthorservices.editage.com/open-access/hybrid.html.
Please note that if your article is accepted, you will receive an email from the editorial office asking you to choose a publication route (traditional or open access). Please keep an eye out for that future email and be sure to respond to it promptly.
***
If you choose to revise your manuscript, please submit your revision through Editorial Manager at
http://ong.editorialmanager.com. Your manuscript should be uploaded in a word processing format such as Microsoft Word.
Your revision's cover letter should include the following:
* A confirmation that you have read the Instructions for Authors (http://edmgr.ovid.com/ong/accounts/authors.pdf), and
* A point-by-point response to each of the received comments in this letter. Do not omit your responses to the Editorial Office or Editors' comments.
If you submit a revision, we will assume that it has been developed in consultation with your co-authors and that each author has given approval to the final form of the revision.
Again, your paper will be maintained in active status for 21 days from the date of this letter. If we have not heard from you by Feb 19, 2021, we will assume you wish to withdraw the manuscript from further consideration.
Sincerely,
John O. Schorge, MD Associate Editor, Gynecology
2019 IMPACT FACTOR: 5.524
2019 IMPACT FACTOR RANKING: 6th out of 82 ob/gyn journals
View Letter ..
5 of 6 2/16/2021, 12:46 PM
__________________________________________________
In compliance with data protection regulations, you may request that we remove your personal registration details at any time. (Use the following URL: https://www.editorialmanager.com/ong/login.asp?a=r). Please contact the publication office if you have any questions.
View Letter ..
6 of 6 2/16/2021, 12:46 PM
Page 1 of 16 February 14, 2021
Dear Dr. Rouse, Editor-in-Chief:
The authors thank you, the reviewers, and the statistics editor for the thoughtful feedback and recommendations regarding our manuscript entitled “Reputational risk: a randomized trial of how a resident’s reputation impacts laparoscopic skills assessment”. We have made the recommended changes to the manuscript and request that it be considered for publication in its updated form.
Please see our detailed response to each of the comments, questions, and recommendations beginning on Page 3 of this letter.
Sincerely,
Name: Dr. Evan Tannenbaum - *The manuscript’s guarantor.
December 11, 2020
Dear Dr. Rouse, Editor-in-Chief:
Please find enclosed our manuscript entitled “Reputational risk: a randomized trial of how a resident’s reputation impacts laparoscopic skills assessment” in consideration for publication in the journal Obstetrics & Gynecology. This study involves the evaluation of bias in the assessment of residents’
surgical skills and we believe it would make a meaningful contribution to your Special Focus: Bias in the Ob-Gyn Workplace collection. We intend to submit solely to this journal and verify that the manuscript is not under consideration elsewhere and will not be submitted elsewhere unless a final negative decision is made by the Editors of Obstetrics & Gynecology.
The lead author* affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This study is a randomized trial, though it does not involve therapeutic or medical behavioural outcomes, rather it characterizes the impact of bias on resident assessment. Therefore, we did not
Page 2 of 16 include a CONSORT checklist, though our methodological rigor is comparable to CONSORT standards.
Also, we did not register this study with ClinicalTrials.gov or another comparable site, as there was no enrollment of patients in the study. The videos that we used were selected from a pre-existing library of available footage, for which consent for use in education and research had been obtained, per
institutional protocol at the time. Our study is not a randomized controlled trial in the traditional sense of the phrase, as it does not involve a medical or behavioral intervention related to a health outcome.
Rather, we used participant randomization to control for confounding and isolate the effect that bias about a resident's reputation has on faculty OBGYNs when they assess that resident. No patients were directly involved in our study. We reviewed the criteria for inclusion in ClinicalTrials.gov and we did not meet criteria for registration, according to the guidelines on the website. Similarly, regarding reporting guidelines, we did not meet criteria for following strict CONSORT guidelines, though we did use it to structure our approach to developing our methodology and drafting our manuscript. These choices were deliberate to avoid causing any potential confusion amongst readers (i.e. "Wait, is this a clinical trial involving patients? Why would it be registered in ClinicalTrials.gov?").
Research ethics board (REB) approval was obtained from University of Toronto prior to recruitment (REB# 37147). Permission has been obtained from all individuals named in the Acknowledgments section.
This work has not been presented at a research meeting nor posted on a pre-print server. This is the first academic submission.
Many thanks in advance for your consideration.
Sincerely,
Name: Dr. Evan Tannenbaum - *The manuscript’s guarantor.
Page 3 of 16 REVIEWER COMMENTS:
The authors confirm that we have read the Instructions for Authors:
http://edmgr.ovid.com/ong/accounts/authors.pdf
We have consulted with a biostatistician at the Biostatistics Research Unit (BRU), at University Health Network-Sinai Health System in Toronto (thebru.ca), to best address the concerns from the statistics editor. Below is a summary of the response to the reviewer comments and questions in bold. The locations of the relevant corrections have been indicated, according to line number.
Reviewer #1:
Well written and researched paper on evaluation of a trainee surgical skills. The introduction of bias in evaluation is important and readers of the journal will be aided by this research. Use of video evaluation of procedures may mitigate bias as you point out.
Questions: Is the Likert scale appropriate? We chose a Likert-type scale for the global assessment scale. While there is some debate in the literature regarding the optimal number of levels (i.e., 5 vs.
7), it is generally accepted that a five-point global assessment scale is appropriate for an overall judgement. This type of scale is similar to the overall performance scale used in our competency- based assessment residency program in Canada, which is similar to the American “Milestones Project”. The competency-based scales contain five levels of “entrustment” (i.e., “do I, the faculty, trust you, the resident, to perform this entire procedure independently?”). We chose more generic anchors (e.g., needs improvement, at expected level, etc.) because the assessors cannot judge things like pre-op assessments or communication skills in the OR that are key features in the entrustment decision-making process.
Page 4 of 16 There seems to be overlap in SD in all groups, can you address? This is correct and we have included our post-hoc analysis to show that the pairwise comparisons indicate a significant difference between the low and the high groups, but not between the low and average or average and high groups in both the primary and secondary outcomes. Also note the updated statistical tests used (non-parametric vs.
previous parametric).
Could you expand on other surgical specialties that have done similar research? We have expanded on this in the introduction: video-based assessment of surgical skills has been studied in general surgery and urology.
Reviewer #2:
Precis - faxulty assessment of residents is influenced by assessor's knowledge of past performance Abstract - Objective - quantify effect of resident's reputation on assessment skills
Methods - faculty randomized to 1 of 3 hypothetical resident scenarios - resident with high, average, or low surgical skills
view video of same resident performing l/s salpingoooporectomy and assessment with OSATS Results - assessment scores on OSATS (out of 20) and on global score (out of 5) varied by reputation:
high resident - 14.8/3.7, average 13.5/3.4, low - 11.2/2.9
Conclusion - assessment is influenced by knowledge of past performance and knowledge introduces bias that affects scores
Intro - work place- based assessments used but are subject to bias - unreliable and unfair data; to mitigate bias - measure bias - cultural and systems changes to mitigate bias
Page 5 of 16 video based assessments to quantify effect of reputation as form of bias
Methods - multi-center, single blind, random study - high, average or remedial group, conducted at 9 sites with video of surgery
stated purpose to assessors was to explore the reliability of OSATS so they wouldn't be biased in assessment ; 8 faculty needed per group Please see the revised sample size calculation in the methods section, as well as our response to the Statistical Editor’s comments and guidance, below.
Results - 43 observers - 14.8/13.5/11.3 and 3.7/3.4/2.9 for the 2 scores
Discussion - bias exists in workplace based assessments and reputation affects assessments
need fair and accurate assessment data to detect underperformance of those with good reputation and to allow appropriate evaluation of those with remedial reputation
data is needed to support the validity of video based assessments but reputation affects assessment so objective assessment is needed
Comments -
This is an interesting study on the impact of bias - the findings are concerning, though not at all surprising.
I imagine the same findings could be found with residents who are well liked versus not as well liked and this would be a tool to assess that.
I would include some data as to what is considered to be a significant variation for the OSATS or the global scale - though these numbers are statistically significant, what constitutes a meaningful difference in assessment? This is a difficult question to answer directly. We demonstrated a 17% difference
Page 6 of 16 (3.5/20) in mean total modified OSATS scores between the high- and low-performing groups for the exact same video performance. When we consider the range-restriction that we see in practice (i.e., most residents receiving scores of 3-5/5. And really, the vast majority receiving 4/5), such a difference for the exact same video appears “clinically” or “educationally” significant. Similarly, the data from the global assessment scale show that 5/14 (35.7%) of faculty failed the low-performing resident, giving them a score of 2/5, compared to the average (1/15, 7.1%) and high (1/14, 6.7%) residents. So, the chances of being failed appear to be higher if you have a reputation as being a low-performer. We can compare these results to those of Aggarwal et al. (reference 23), who compared modified OSATS scores for laparoscopic cholecystectomies performed by novice- and expert-surgeons. These authors demonstrated a non-significant difference between those groups (14 vs. 15 out of 20, respectively, p>0.05). When we compare those results to ours, we can see that the difference of 3.5/20 points is actually quite striking, especially considering the respondents were looking at the exact same video.
Our apologies for the long-winded answer, but in sum, while there is no established threshold for educational or clinical significance that we are aware of, compared to previous studies that explored differences between expert and novice surgeons, using the same assessment scale as in our study, we showed a larger difference in scores based on reputation, irrespective of actual performance.
Similarly, the chances of being “failed” are higher for those reputed to be low-performers. This has been explained in the discussion section.
Reviewer #3:
This manuscript explores a type of unconscious bias in the evaluation of learners in Ob-Gyn residency.
Namely, the impact of resident reputation on assessment of laparoscopic skills using what was designed
Page 7 of 16 to be an "objective" evaluation (OSATS). Through a simple, but elegant design, the investigators
highlight a potential source of bias even using the OSAT tool that by design is meant to be a
standardized approach to resident assessment. The idea of implicit bias is not new, but the results of this study highlight a critical area of concern in the evaluation of learners even when using objective and standardized tools. Thank you.
STATISTICS EDITOR COMMENTS:
Lines 139-141: How many observers rated the videos and was the 3 of 5 score an average, or a consistent grade among all observers? Need to clarify re: the video used for the rest of the study. We had the video assessed by a total of six people: three of the investigators (ET, MS, MW), and three minimally-invasive surgery fellows at our institution. We decided on the final video based on an analysis of the average score for each of the four scale items. We selected the video that was scored on average 12/20 total possible points with the least variability between items (i.e., the closest to 3/5 x 4 items). We did not calculate consistency or inter-rater reliability (e.g., with ICC) during the pilot phase because not all raters in the piloting phase viewed multiple videos. This is reflected in the revised wording.
lines 166-172: The sample size calculation needs clarification. It appears that the Authors were
stipulating a 2 point difference in means, with a SD = 2, equal cohort sizes, power = 80% and alpha = .05.
In order to compare two groups, the required sample size = 17 in each group. The ANOVA test would test whether the overall data distribution did/did not conform to no difference, but it would not test specific pairs. It appears that the Authors were positing that the high, average and low performing
Page 8 of 16 groups would differ by increments of 2 in their mean values, which is not what was tested by ANOVA.
Furthermore, the sample size does not allow simultaneous testing of the 5 scale Likert comparison.
Sample size calculation: We agree that the sample size calculation provided was inaccurate, and was not meant to mislead. Rather, it reflected an error on our part. Unfortunately, we are not able to collect any additional data, as the use of deception has been disclosed and debriefed with
participants, per our institutional policy. Therefore, it was suggested that we proceed with a power calculation using the current sample size. The power calculation was performed in R and we have attached the R-package we used as a PDF. With a sample size of 43, assuming alpha of 0.05, we are able to detect, with 80% power, the smallest difference equal to 2.5 points on the total modified OSATS scale among the three groups, assuming a standard deviation of 2. See revised sample size calculation in the methods section.
Regarding the use of ANOVA: We have updated the statistical analyses, using non-parametric tests and post-hoc pairwise testing. We demonstrate a significant difference between the low and high- performing groups but not between the low and average, or average and high. Considering the aim of the study is to explore the effect of bias against low-performers and illustrate the potential harm to those undergoing remediation, we believe these results are meaningful for the reader.
The 5 point Likert scale should have been formatted as median(range or IQR) and tested non- parametrically and presented as secondary outcome.
We agree and have included the updated analyses in the abstract, methods, and results sections.
Given the sample sizes, the OSATS should also be statistically evaluated using a non-parametric test (Kruskal-Wallis) for the overall test with pair wise testing or using one group as the referent for testing between groups.
Page 9 of 16 We replaced the ANOVAs with non-parametric tests. The updated analyses are reflected in the
abstract as well, where median and IQR values have replaced the means.
Fig 1: See comments on use of means to summarize the groups. We have replaced the figures with boxplots (see updated Figure 1 and 2 PDF files).
EDITORIAL OFFICE COMMENTS:
1. The Editors of Obstetrics & Gynecology are seeking to increase transparency around its peer-review process, in line with efforts to do so in international biomedical peer review publishing. If your article is accepted, we will be posting this revision letter as supplemental digital content to the published article online. Additionally, unless you choose to opt out, we will also be including your point-by-point response to the revision letter. If you opt out of including your response, only the revision letter will be posted.
Please reply to this letter with one of two responses:
A. OPT-IN: Yes, please publish my point-by-point response letter. Please include the point-by-point response letter.
B. OPT-OUT: No, please do not publish my point-by-point response letter.
2. Obstetrics & Gynecology uses an "electronic Copyright Transfer Agreement" (eCTA). When you are ready to revise your manuscript, you will be prompted in Editorial Manager (EM) to click on "Revise Submission." Doing so will launch the resubmission process, and you will be walked through the various questions that comprise the eCTA. Each of your coauthors will receive an email from the system
requesting that they review and electronically sign the eCTA.
Please check with your coauthors to confirm that the disclosures listed in their eCTA forms are correctly disclosed on the manuscript's title page.
Page 10 of 16 3. Clinical trials submitted to the journal as of July 1, 2018, must include a data sharing statement. The statement should indicate 1) whether individual deidentified participant data (including data
dictionaries) will be shared; 2) what data in particular will be shared; 3) whether additional, related documents will be available (eg, study protocol, statistical analysis plan, etc.); 4) when the data will become available and for how long; and 5) by what access criteria data will be shared (including with whom, for what types of analyses, and by what mechanism). Responses to the five bullet points should be provided in a box at the end of the article (after the References section).
Not applicable
4. Responsible reporting of research studies, which includes a complete, transparent, accurate and timely account of what was done and what was found during a research study, is an integral part of good research and publication practice and not an optional extra. Obstetrics & Gynecology supports initiatives aimed at improving the reporting of health research, and we ask authors to follow specific guidelines for reporting randomized controlled trials (ie, CONSORT). Include the appropriate checklist for your manuscript type upon submission. Please write or insert the page numbers where each item appears in the margin of the checklist. Further information and links to the checklists are available at http://ong.editorialmanager.com. In your cover letter, be sure to indicate that you have followed the CONSORT, MOOSE, PRISMA, PRISMA for harms, STARD, STROBE, RECORD, CHEERS, SQUIRE 2.0, or CHERRIES guidelines, as appropriate.
Not applicable
5. Standard obstetric and gynecology data definitions have been developed through the reVITALize initiative, which was convened by the American College of Obstetricians and Gynecologists and the members of the Women's Health Registry Alliance. Obstetrics & Gynecology has adopted the use of the
Page 11 of 16 reVITALize definitions. Please access the obstetric data definitions at https://www.acog.org/practice- management/health-it-and-clinical-informatics/revitalize-obstetrics-data-definitions and the gynecology data definitions at https://www.acog.org/practice-management/health-it-and-clinical-
informatics/revitalize-gynecology-data-definitions. If use of the reVITALize definitions is problematic, please discuss this in your point-by-point response to this letter.
Not applicable
6. Because of space limitations, it is important that your revised manuscript adhere to the following length restrictions by manuscript type: Original Research reports should not exceed 22 typed, double- spaced pages (5,500 words). Stated page limits include all numbered pages in a manuscript (i.e., title page, précis, abstract, text, references, tables, boxes, figure legends, and print appendixes) but exclude references.
Confirmed
7. Specific rules govern the use of acknowledgments in the journal. Please note the following guidelines:
* All financial support of the study must be acknowledged.
* Any and all manuscript preparation assistance, including but not limited to topic development, data collection, analysis, writing, or editorial assistance, must be disclosed in the acknowledgments. Such acknowledgments must identify the entities that provided and paid for this assistance, whether directly or indirectly.
* All persons who contributed to the work reported in the manuscript, but not sufficiently to be authors, must be acknowledged. Written permission must be obtained from all individuals named in the
acknowledgments, as readers may infer their endorsement of the data and conclusions. Please note that
Page 12 of 16 your response in the journal's electronic author form verifies that permission has been obtained from all named persons.
* If all or part of the paper was presented at the Annual Clinical and Scientific Meeting of the American College of Obstetricians and Gynecologists or at any other organizational meeting, that presentation should be noted (include the exact dates and location of the meeting).
Confirmed
8. The most common deficiency in revised manuscripts involves the abstract. Be sure there are no inconsistencies between the Abstract and the manuscript, and that the Abstract has a clear conclusion statement based on the results found in the paper. Make sure that the abstract does not contain information that does not appear in the body text. If you submit a revision, please check the abstract carefully.
Confirmed
In addition, the abstract length should follow journal guidelines. The word limit for Original Research articles is 300 words; Reviews is 300 words; Case Reports is 125 words; Current Commentary articles is 250 words; Executive Summaries, Consensus Statements, and Guidelines are 250 words; Clinical Practice and Quality is 300 words; Procedures and Instruments is 200 words. Please provide a word count.
Confirmed
9. Abstracts for all randomized, controlled trials should be structured according to the journal's standard format. The Methods section should include the primary outcome and sample size justification. The Results section should begin with the dates of enrollment to the study, a description of demographics,
Page 13 of 16 and the primary outcome analysis. Please review the sample abstract that is located online here:
http://edmgr.ovid.com/ong/accounts/sampleabstract_RCT.pdf. Please edit your abstract as needed.
Confirmed
10. Only standard abbreviations and acronyms are allowed. A selected list is available online at
http://edmgr.ovid.com/ong/accounts/abbreviations.pdf. Abbreviations and acronyms cannot be used in the title or précis. Abbreviations and acronyms must be spelled out the first time they are used in the abstract and again in the body of the manuscript.
Confirmed
11. The journal does not use the virgule symbol (/) in sentences with words. Please rephrase your text to avoid using "and/or," or similar constructions throughout the text. You may retain this symbol if you are using it to express data or a measurement.
Confirmed
12. In your Abstract, manuscript Results sections, and tables, the preferred citation should be in terms of an effect size, such as odds ratio or relative risk or the mean difference of a variable between two groups, expressed with appropriate confidence intervals. When such syntax is used, the P value has only secondary importance and often can be omitted or noted as footnotes in a Table format. Putting the results in the form of an effect size makes the result of the statistical test more clinically relevant and gives better context than citing P values alone.
If appropriate, please include number needed to treat for benefits (NNTb) or harm (NNTh). When comparing two procedures, please express the outcome of the comparison in U.S. dollar amounts.
Page 14 of 16 Please standardize the presentation of your data throughout the manuscript submission. For P values, do not exceed three decimal places (for example, "P = .001"). For percentages, do not exceed one decimal place (for example, 11.1%").
Confirmed
13. Please review the journal's Table Checklist to make sure that your tables conform to journal style.
The Table Checklist is available online here: http://edmgr.ovid.com/ong/accounts/table_checklist.pdf.
Confirmed
14. Please review examples of our current reference style at http://ong.editorialmanager.com (click on the Home button in the Menu bar and then "Reference Formatting Instructions" document under "Files and Resources). Include the digital object identifier (DOI) with any journal article references and an accessed date with website references. Unpublished data, in-press items, personal communications, letters to the editor, theses, package inserts, submissions, meeting presentations, and abstracts may be included in the text but not in the reference list.
In addition, the American College of Obstetricians and Gynecologists' (ACOG) documents are frequently updated. These documents may be withdrawn and replaced with newer, revised versions. If you cite ACOG documents in your manuscript, be sure the reference you are citing is still current and available. If the reference you are citing has been updated (ie, replaced by a newer version), please ensure that the new version supports whatever statement you are making in your manuscript and then update your reference list accordingly (exceptions could include manuscripts that address items of historical interest). If the reference you are citing has been withdrawn with no clear replacement, please contact the editorial office for assistance ([email protected]). In most cases, if an ACOG document has
Page 15 of 16 been withdrawn, it should not be referenced in your manuscript (exceptions could include manuscripts that address items of historical interest). All ACOG documents (eg, Committee
Opinions and Practice Bulletins) may be found at the Clinical Guidance page at https://www.acog.org/clinical (click on "Clinical Guidance" at the top).
Confirmed
15. Figures 1-2: okay (see below)
When you submit your revision, art saved in a digital format should accompany it. If your figure was created in Microsoft Word, Microsoft Excel, or Microsoft PowerPoint formats, please submit your original source file. Image files should not be copied and pasted into Microsoft Word or Microsoft PowerPoint.
When you submit your revision, art saved in a digital format should accompany it. Please upload each figure as a separate file to Editorial Manager (do not embed the figure in your manuscript file).
If the figures were created using a statistical program (eg, STATA, SPSS, SAS), please submit PDF or EPS files generated directly from the statistical program.
Figures should be saved as high-resolution TIFF files. The minimum requirements for resolution are 300 dpi for color or black and white photographs, and 600 dpi for images containing a photograph with text labeling or thin lines.
Page 16 of 16 Art that is low resolution, digitized, adapted from slides, or downloaded from the Internet may not reproduce.
We have updated the figures, per the guidance from the statistics editor. These updated files were exported directly from SPSS as PDFs and are attached in the submission materials marked
“REVISIONS”.
16. Authors whose manuscripts have been accepted for publication have the option to pay an article processing charge and publish open access. With this choice, articles are made freely available online immediately upon publication. An information sheet is available at http://links.lww.com/LWW-ES/A48.
The cost for publishing an article as open access can be found at https://wkauthorservices.editage.com/open-access/hybrid.html.
Please note that if your article is accepted, you will receive an email from the editorial office asking you to choose a publication route (traditional or open access). Please keep an eye out for that future email and be sure to respond to it promptly.
