Like many in the clinical practice guideline community, the team working on the Agency for Healthcare Research and Quality’s (AHRQ’s) National Guideline Clearinghouse (NGC), www.guideline.gov, devoured the Institute of Medicine’s (IOM’s) report on trustworthy guidelines when it was released in 2011[i]. For years leading up to the report, NGC and AHRQ staff and leaders had been hearing that there were too many guidelines of varying quality in NGC. In fact, others had been hearing this also and, in 2008, the U.S. Congress requested that the Secretary of the Department of Health and Human Services (DHHS) contract with the IOM to identify "the best methods used in developing clinical practice guidelines in order to ensure that organizations developing such guidelines have information on approaches that are objective, scientifically valid, and consistent."[ii] AHRQ funded the IOM contract.

The IOM guideline report and its sister report on the standards of systematic reviews launched a sequence of steps AHRQ and the NGC team have taken to support dissemination of evidence-based guidelines developed in response to the IOM report.

One of those steps involved adopting the 2011 IOM revised definition of a clinical practice guideline and revamping the criteria for inclusion of guidelines that met the definition. Analysis, input, prototyping, vetting and refining the inclusion criteria were key activities leading up to the announcement of NGC’s revised criteria in June 2013 and decision to begin implementing the criteria in June 2014.

The major change in the inclusion criteria centered on the evidence-basis underpinning the clinical practice guidelines; specifically, that guidelines are based on a systematic evidence review. While the original 1997 inclusion criteria had a criterion related to systematic reviews, the 2013 (revised) criterion provided greater specificity that clarified expectations about the systematic reviews used in the development of the guidelines. See related text box.

NGC’s Evidence (Systematic Review) Criterion: The clinical practice guideline is based on a systematic review of evidence as demonstrated by documentation of each of the following features in the clinical practice guideline or its supporting documents. An explicit statement that the clinical practice guideline was based on a systematic review. A description of the search strategy that includes a listing of database(s) searched, a summary of search terms used, the specific time period covered by the literature search including the beginning date (month/year) and end date (month/year), and the date(s) when the literature search was done. A description of study selection that includes the number of studies identified, the number of studies included, and a summary of inclusion and exclusion criteria. A synthesis of evidence from the selected studies, e.g., a detailed description or evidence tables. A summary of the evidence synthesis (see d above) included in the guideline that relates the evidence to the recommendations, e.g., a descriptive summary or summary tables. NB: A guideline is not excluded from NGC if a systematic review was conducted that identifies specific gaps in the evidence base for some of the guideline's recommendations.

The 2013 inclusion criteria also included a new criterion regarding assessing benefits and harms of recommended and alternative care options. For those familiar with robust systematic review methodology and reporting, this criterion is logical and consistent with the purpose of systematic evidence reviews. For those less familiar, systematic reviews find, assess, synthesize, and interpret the evidence of care options (e.g., treatment interventions) looking at benefits and harms of those options to answer specific clinical care questions.

It’s now one year post-NGC-implementation of the 2013 (revised) inclusion criteria and a perfect opportunity to comment on what a difference one year makes. Take a look:

There has been a significant drop in the number of guidelines being reviewed for inclusion. Many guideline developers report that they continue to flesh out the changes needed in their guideline development and reporting processes in order to meet the 2013 criteria. Some developers have stated that they cannot meet the 2013 criteria and choose to cease participating in NGC, or reduce submissions for other reasons.

Many of the guidelines reviewed for inclusion in the past year did not meet the 2013 criteria. Most of those lacked a systematic evidence review. Specifically, the sub-criterion not met most frequently was for a synthesis of evidence (in the form of evidence tables or a detailed description relating how the guideline developers evaluated individual studies and how the body of evidence answers the key clinical questions).

Because fewer guidelines are meeting the 2013 criteria, fewer NGC guideline summaries are being posted to the NGC site. The team has not been resting on their laurels, however. Instead, they shifted their focus from producing guideline summaries to reviewing guidelines for inclusion and providing related education and assistance to guideline developers. We’ve spent many hours working with individual guideline developers through email and phone conferences to assess, on a case-by-case basis, how a particular guideline would fare against the 2013 inclusion criteria. This has helped developers identify gaps in their process and/or reporting and understand what additional information may be needed for the guideline to meet the criteria. Providing this extra effort to assist guideline developers in the earlier stages of the NGC process has been fruitful; there are early indications that the number of submissions in the upcoming year (i.e., 2-years post implementation) may be slightly higher and will meet 2013 criteria.

The 2013 criteria, these data and our collaborative experiences with participating guideline developers have implications for updating guidelines. Most notably, in 2015, 486 guideline summaries now posted to NGC are in danger of ‘aging out’, meaning they will have reached the 5-year dating criterion and can no longer be retained. In order to continue to be included, most guideline developers will need to submit updated guidelines that meet the 2013 criteria. When looking ahead to the end of 2015, there is a solid chance that a large proportion of these guideline summaries will ‘age out,’ be withdrawn from the site and moved to the archive, lowering the total number of guidelines represented in NGC significantly. As of this writing, there are 2403 guideline summaries in NGC; removing 486 right now would be a 20% reduction.

A decreased number of clinical practice guidelines included in NGC would not be  a surprise and is consistent with the AHRQ NGC’s goal of only including guidelines meeting the 2011 IOM definition of an evidence-based clinical practice guideline. The bar for inclusion was raised. This first year of implementation has identified challenges faced by the guideline developer community in meeting the higher bar. These challenges are of great interest to AHRQ, an agency on a mission to assure that evidence of higher quality health care is produced, understood and used. To help fulfil that mission, the NGC team is available to clinical practice guideline developers who are developing or updating their guidelines for pre-submission assessments. AHRQ encourages guideline developers to connect with the NGC team and tap into existing resources for support.