The Institute of Medicine Standards for Systematic Reviews: 4 Years Later

September 15, 2015
SallyMorton

By Sally C. Morton, PhD
Professor and Chair, Department of Biostatistics University of Pittsburgh
Vice Chair, IOM Committee on Standards for Systematic Reviews of Comparative Effectiveness Research

Introduction

In May 2011, two companion reports from the Institute of Medicine (IOM) were published on standards for systematic reviews (SRs)[1] and clinical practice guidelines (CPGs)[2] respectively. The IOM CPG Report noted the fundamental importance of SRs in its definition of CPGs as “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence.”[3] Furthermore, the IOM CPG Report’s Standard 4.1 states “Clinical practice guideline developers should use systematic reviews that meet standards set by the Institute of Medicine’s Committee on Standards for Systematic Reviews of Comparative Effectiveness Research.”[4] Much recent commentary on this website has concerned CPGs, and so it is appropriate to ask what has transpired since the IOM SR Report.
I would be remiss not to mention the important vehicle for education and influence that the Patient-Centered Outcomes Research Institute (PCORI) provides for the IOM SR standards. PCORI, as directed in its enabling legislation, has established 47 methodological standards for the conduct of patient-centered comparative effectiveness research.[5] PCORI requires investigators requesting funding to demonstrate how they would follow all relevant standards in their proposals, and if funded, investigators must demonstrate adherence to the standards. Two PCORI standards include the IOM SR standards, one directly, and one indirectly. The first is PCORI standard SR-1, which states that if one is conducting an SR, the IOM SR standards should be followed except for some qualifications, namely that including non-English sources is not recommended except for certain clinical topics, dual screening is desirable but fact-checking may suffice, and independent librarian review is not required. These qualifications underscore a necessity for flexibility based on the clinical topic being studied, and the resources and time available.

The PCORI standard that indirectly includes the IOM SR standards is RQ-1, which concerns identifying gaps in evidence. This standard requires that if an SR on gaps has not been conducted, researchers should perform one according to the PCORI standard SR-1, or give sufficient reason as to why an SR is not possible. It is important to note that all PCORI projects, regardless of purpose or design, must follow the RQ-1 standard, which demonstrates the wide impact the IOM SR report can have.

Influence of IOM Report

To further examine the influence of the IOM SR Report, a recent search[6] of Scopus and Web of Science was conducted, revealing 257 citations of the report since publication.  Seventy-six (30%) of these references were on specific guidelines, and 58 of those were American College of Cardiology (ACC)/American Heart Association (AHA) guidelines, with a single guideline often repeated in different journals. The ACC/AHA, a leading producer of guidelines, updated the SR methodology in their guideline development process specifically to address the IOM SR standards. The remaining 18 of 76 guideline references were on individual guidelines by a wide array of institutions and concerning a variety of conditions.


Twenty-four (9%) of the 257 references were on methods for guidelines, and 41 (16%) were individual SRs. Forty (15%) were on various other topics, the majority of which were discussions of evidence-based medicine in various clinical settings.
The remaining 76 (30%) references were on methods for SRs, and are perhaps the most relevant for this discussion, as they indicate the influence of the IOM SR Report on the science of SRs. As background, it is important to recall several key characteristics of the IOM SR Report. First, the report’s overall objective was to provide standards for “comprehensive, publicly funded SRs”[7] of comparative effectiveness research, in essence the gold standard of SRs. Second, the report recommended 21 standards with 82 elements of performance, and admitted that “Collectively the standards and elements present a daunting task. Few, if any, members of the committee have participated in an SR that fully meets all of them. Yet the evidence and experience are strong enough that it is impossible to ignore these standards or hope that one can safely cut corners.”[8] Third, the report did not focus on promoting specific methodology, for example particular meta-analytic techniques, but rather on providing “the building blocks that help ensure objective, transparency, and scientific rigor,”[9] so that methodological innovation would flourish. Good examples of this stance are the standard and element concerning the decision to conduct a meta-analysis, which are fundamental rather than technical:[10]


Among the 76 references on SR methods is Chang et al.,[11] a response from the Agency for Healthcare Research and Quality (AHRQ) Evidence-Based Practice Center (EPC) Program to the IOM SR Report. These authors underscore the aspirational aspect of the report, and provide advice on how to meet the standards given limited time and resources. They also counsel flexibility and efficiency, which is echoed in the PCORI SR-1 adaption of the IOM standards, and additional references demonstrate the considerable contribution of the EPC Program to SR methods.[12] Of course, the Cochrane Collaboration, one of the expert guidance sources utilized in the IOM SR Report, is a leader in methodological contributions as well, and several of the 76 references on SR methods emanate from Cochrane collaborators or utilize Cochrane resources.

Conclusion

Several of the 76 references are focused on particular steps in an SR, foreshadowed by the IOM SR Report’s identification of open questions. Extremely topical methodological and procedural challenges such as automatic data extraction,[13] complex interventions,[14] language restriction,[15] and screening approaches[16] are addressed, though network meta-analysis,[17] which is very relevant for SRs of comparative effectiveness research, is not. Suffice to say that the report’s goal of setting expectations with accountability without stifling innovation has been realized. However, despite the positive influence of the IOM SR Report, I must quote my colleague Dr. Greenfield, whose Commentary[18] concerned progress since the IOM CPG report: “The stakes are high and much remains to be done.  The public is counting on us all.” I agree completely.

References

  1. IOM (Institute of Medicine). 2011. Finding what works in health care: Standards for systematic reviews. Eden J, Levit L, Berg A, Morton S (editors). (“SR Report”) Available at http://www.nap.edu/catalog/13059/finding-what-works-in-health-care-standards-for-systematic-reviews.  Accessed September 9, 2015.
  2. IOM (Institute of Medicine). 2011. Clinical practice guidelines we can trust. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E (editors). (“CPG Report”). Available at http://www.nap.edu/catalog/13058/clinical-practice-guidelines-we-can-trust.  Accessed September 9, 2015.
  3. IOM CPG Report 2011, page 4.
  4. IOM CPG Report 2011, page 7.
  5. PCORI (Patient-Centered Outcomes Research Institute) Methodology Committee. 2013. The PCORI Methodology Report. Available at http://www.pcori.org/research-results/research-methodology. Accessed September 9, 2015.
  6. Search conducted on September 3, 2015.
  7. IOM SR Report 2011, page 3
  8. IOM SR Report 2011, page 4.
  9. IOM SR Report, 2011, page 156.
  10. IOM SR Report, 2011, page 159.
  11. Chang SM, Bass EB, Berkman N, Carey TS, Kane RL, Lau J, Raticheck S. Challenges in implementing The Institute of Medicine systematic review standards. Systematic Reviews 2013;2:69.
  12. Chang SM. The Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program Methods Guide for comparative effectiveness reviews: Keeping up-to-date in a rapidly evolving field. JCE 2011;64:1166-1167.
  13. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: A systematic review. Systematic Reviews 2015;4:78.
  14. Squires JE, Valentine JC, Grimshaw JM. Systematic reviews of complex interventions: Framing the review question. JCE 2013;66:1215-1222.
  15. Wang Z, Brito JP, Tsapas A, Griebeler ML, Alahdab D, Murad MH. Systematic reviews with language restrictions and no author contact have lower overall credibility: A methodology study. Clin Epi 2015;7:243-247.
  16. Mateen F, Oh J, Tergas AI, Nrhayani NH, Kamdar NN. Titles versus titles and abstracts for initial screening of articles for systematic reviews. Clin Epi 2013;5:89-95.
  17. Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JPT. Evaluating the quality of evidence from a network meta-analysis. PLOS ONE 2014:9(7);e99682.
  18. Greenfield S. The Institute of Medicine Standards for Trustworthy Clinical Practice Guidelines: 4 Years Later. May 2015. Accessed September 9, 2015.