Supplemental Digital Content 1. Detailed Methodology and Risk of Bias
Protocol development
A rapid review was conducted according to Dobbins as well as Moher et al (the PRISMA statement).3,4 The review progressed according to multiple phases, beginning with the development of the research protocol.
This protocol included the generation of a primary research question and a search strategy aimed at discovering materials commensurate to that question. The protocol also included criteria for screening citations and abstracts, appraising full-text eligibility and data collection and summarization.
Citations were all obtained using “Publish or Perish” (PoP) Windows software based within Google Scholar. This was beneficial for both finding peer-reviewed materials and gray literature. Searches were performed via two separate methods: 1) works cited by the 2012 IOM report through multiple iterations and 2) keyword search to obtain results for each of the 10 recommendations. Using the “Cited Works”
feature in PoP, starting with the IOM report as a base level, a new level of works citing the prior level were generated. This continued for several more levels, expanding outward. For instance, the IOM report was cited by 47 articles at the time of search; the second iteration of citations found articles which cited at least one of those 47 articles, and so forth. Next, ten separate keyword search strategies were used, one for each of the 10 IOM recommendations. Each of the 10 search strategies were refined to obtain a total of 100-200 results for each strategy using 1-2 keyword searches. The keyword searches and Cited Works strategies were performed between May 2 and May 6, 2019 and all citation results were aggregated together.
Eligibility criteria
Duplicate citations were first removed so that only unique citations remained. Next, citation titles and abstracts were screened for relevance to the research question. Relevant citations were retained and irrelevant citations were removed. Then, the full-text materials of relevant citations were obtained for eligibility review. Materials were eligible for inclusion if they (1) addressed the topic of at least one of the recommendations and (2) contained information of either official action taken or evidence influence to the practice community. For example, material referencing a “minimum package of public health services” and
describing impacts of its implementation within a state would be eligible. Types of articles reviewed for eligibility included both original peer-reviewed literature (cross-sectional, longitudinal and reviews), commentaries, analytic essays and gray literature (reports, proceedings and other publications). Only materials in English and descriptive of the United States or its territories were considered and, as the final IOM report was published in 2012, only materials published between January 2012 and May 2019 were accepted.
Search and study selection
As the IOM report contained ten distinct recommendations, each recommendation was treated as a separate sub-question of the main research question for the keyword searches. The advanced search function was leveraged to craft individual searches narrowed to different facets of each recommendation. Commonalities among keywords existed between most searches with many sharing the use of “Institute of Medicine”,
“public health”, “minimum package” and similar terms; the full list of search terms are available within a methodology addendum (see Table, Supplemental Digital Content 1, which lists citation metrics per search). Two to three searches were performed for each recommendation with aims to yield one hundred or more “hits” per recommendation.
Data collection
Each citation returned from the search strategy via Google Scholar was entered into a spreadsheet. The citation included metadata fields such as title, author(s), year published, URL (if available) and other similar fields. The data collection spreadsheet contained additional columns to 1) recommend screening disposition (yes/no) and eligibility (yes/no); 2) mark whether one or more recommendations were discussed within material; 3) enter details on research (methods, results, discussion); 4) influence of 2012 IOM report (report reception, evolution, disposition); and 5) any key quotes from the material. Information was added to the spreadsheet after each document was reviewed.
Once all documents were reviewed, data collected from eligible materials was reviewed to identify themes from research and reception of the IOM report. Data were categorized within the following buckets:
Discussion of IOM Report, Research Study Overviews, Utilization of IOM Report and Quotes. Within the Discussion, Utilization and Quotes buckets, items were further categorized by individual recommendation while the Overviews were summarized by common themes among recommendations. For example, discussion of public health revenues is common among recommendations 2, 8 and 9 whereas minimum package definition and costing are shared between recommendations 5 and 7.
Risk of Bias in Review and Review Limitations
For the review process, there were several common risks and limitations associated with rapid reviews of literature. Selection bias may exist if only one reviewer selects studies, leading to reduced transparency and reproducibility; two primary reviewers were used with a third acting to resolve conflicts between the primaries. Exclusion of gray literature may lead to a lack of unpublished or negative results as well as other significant results. The search strategy was conducted exclusively within Google Scholar as the search engine’s comprehensive database can identify nearly all citations as similar scholarly search engines (greater than ninety percent) and a substantial amount of unique citation – including a large amount of gray literature – not found by other databases.60-62 However, some activities known by authors to be linked to the publication of the IOM report were not identified through the search strategy and are not present within results but highlighted within the discussion.
Exclusion of older articles typically obscures other significant results though, in this case, relevant materials would only have been published following the March 2012 publication of the IOM report. Lastly, even though a protocol was established in advance of the study and pairs of reviewers were used for both the screening and eligibility review phases, only one of the authors performed data extraction and thematic analysis. Formal quality appraisal was not conducted, however, given the nature of analysis for this review, the stringent screening and eligibility process was deemed sufficient for the review.
Limitations existed from the materials obtained, as well. Though the focus of this review was to understand the official actions taken and impacts to the practice community as a result of the IOM report’s publication, the heterogenous nature of included materials rendered meta-analysis and subgrouping difficult. This was
acceptable, however, as the eligibility review phase allowed for coding according to whether a material covered aspects of one or more recommendation. Though, a challenge persisted in creating a true synthesis of findings from the materials as quantitative findings and qualitative theories were typically not defined or measured in the same way or contained subjective analyses of current paradigms. Any aggregations of research findings or discourse from literature presented within the results were common themes across cited materials.
