Last September, the Commission on Evidence-Based Policymaking released its report, recommending a series of concrete steps to increase the production and use of evidence in government policymaking. Health care programs account for a very substantial share of government spending, so heightening the use of evidence in the design, implementation, and operation of these programs is of obvious importance.
There is, of course, a long history of bringing evidence to bear in the formulation of health policy. The Office of the Assistant Secretary for Planning and Evaluation (ASPE) at the Department of Health and Human Services (HHS) was established in 1966 with the stated purpose of helping the Department (then Health Education and Welfare) better implement programs. Legislative rules and Office of Management and Budget guidance, such as the requirement that agencies conduct Government Performance and Results Act (GPRA) reviews for ongoing programs and perform Regulatory Impact Analyses (RIAs) for new regulations, formally enshrine a role for evidence in government. But the ecosystem in which government evidence-building and evidence-using activities occur has changed tremendously in the time since ASPE was formed, and the GPRA and RIAs were introduced.
Using evidence to improve government is what taxpayers expect- smart and careful use of limited resources to best address national priorities. -Analytical Perspectives, Budget of the United States Government, Fiscal Year 20181
Take one simple example: back in 1978, a standard government-issue desktop computer had about four kilobytes of core memory and could conduct 300,000 operations in a second. Today, a standard issue cellphone costs about ⅕ as much, has about 800 times as much memory, and conducts about 2000 times as many operations each second. Evidence is produced through the analysis of data. Innovations in computing mean that we could, in principle, have thousands of times more data and analysis available for government decision-making now than we did 40 years ago. Our infrastructure for collecting, analyzing, and using data, however, has not kept pace with this technological progress.
Technological improvements offer new opportunities to turn existing administrative data into a resource that guides decision making. We often think about making better use of data when considering big policy choices, like how to expand health insurance coverage across the US population. Government agencies, including the Congressional Budget Office and ASPE, have often used health data for this purpose. Improvements in computing mean they can do this work better. For example, linking data on individual incomes from the Internal Revenue Service to data on health insurance enrollment decisions could substantially improve our estimates of how people will respond to income-based subsidies.
Data are the lifeblood of decision-making and the raw material for accounatbility. Without high-guality data providing the right infomation on the right things at the right time; designing, monitoring and evalutaing effective policies becomes almost impossible. – United Nations Secretary-General’s Independent Expert Advisory Group on a Data Revolution for Sustainable Development2
But some of the biggest technology-driven improvements in the governance of the health care system are likely to come through using our data more routinely, in the context of continuous quality improvements within existing programs. Here, technology offers not only the ability to collect and store data, but the opportunity to use machine learning, A/B testing, low-cost randomized trials, and quasi-experimental methods like regression-discontinuity designs on a routine basis, not just for one-time studies. Imagine if government agencies routinely took advantage of discontinuities generated by scoring algorithms, service interruptions caused by weather disruptions, or alternative formulations of website language to make incremental improvements in the way they operated, just as large technology companies do today. That would require a shift in the evaluation mind-set. Today, agencies formulate questions and then seek data to answer them. In continuous-quality-improvement mode, agencies would regularly look at their data and see what questions they could ask and answer and what improvements they could make with the data they have on hand.
Many of the recommendations of the Commission on Evidence-Based Policymaking would facilitate these changes. One is the recommendation that Congress establish a National Secure Data Service (NSDS). The proposal for development of an NSDS was very much informed by experience with health data, where concerns over both privacy and access have long been important (see Figure 1).
The Commission learned that despite very significant concerns about patient confidentiality, researchers inside and outside government have long had better access to health care data than to many arguably less sensitive government data sources. Paradoxically, one reason for this improved access has been that there is a more robust and well-understood regulatory framework for protecting health data, established through the Health Insurance Portability and Access Act (HIPAA) in 1996. That example taught the Commission that improved privacy and improved access can go hand-in-hand. Promulgating uniform standards limits careless practices that put data at risk and allows establishment of well-understood processes that allow legitimate researchers to use data for statistical purposes (see Figure 2).
Another lesson the Commission learned from the health sphere is that data linkage can be performed effectively without collecting and maintaining a central repository. Storing all federal data in a single monolithic database raises deep and legitimate concerns both about misuse of data by hackers outside government and about government intrusions that limit individual privacy. The NSDS instead is designed as a distributed data model. Researchers can request that a linkage among datasets be made, data can be brought together and combined, and a new linked dataset can be created for statistical analysis. That new dataset can be modified to allow varying levels of individual identification for different releases in more (or less) secure settings and to researchers who are more (or less) trusted. Distributed data resources of this type have been used in a range of healthcare settings, including the Food and Drug Administration’s Sentinel Initiative and the Centers for Medicare and Medicaid Services Edge Server.
The health sector’s experience also informed the Commission’s thinking about how to reorganize government to enhance the routine use of evidence for policymaking and policy implementation (see Figure 3). While ASPE and evaluation offices within the HHS departments have long histories, many government departments and agencies lack a strong evaluation presence. The Commission recommended that every department identify or establish a Chief Evaluation Officer. One role for that officer would be to develop a learning agenda for the Department, identifying key research needs for those within and outside government and helping to develop the evidence base for policy development.
Evidence improves program design and helps existing programs work better. That’s as true in government as in the private sector, where the most successful firms are those that invest in continuous testing of their programs and products. Implementing the recommendations of the Commission on Evidence-Based Policymaking would bring us a big step forward in the development and use of evidence in government. Currently, a bill that would lay the groundwork for implementing the Commission’s recommendations has passed the House and is under consideration in the Senate. But there’s also a lot that agencies like HHS can do today to make progress on improving the use of evidence in its decisions. HHS will soon release a data strategy that articulates priorities for better using their own data, consistent with the commission’s recommendations.
- Office of Management and Budget. Analytical Perspectives, Budget of the U.S. Government, Fiscal Year 2018. Accessed October 10, 2018.
- United Nations, Secretary General’s Independent Expert Advisory Group on a Data Revolution for Sustainable Development. A World That Counts: Mobilizing the Data Revolution for Sustainable Development. New York, NY: United Nations, November 2014. Accessed October 10, 2018.
- Commission on Evidence-Based Policymaking. The Promise of Evidence-Based Policymaking: Report of the Commission on Evidence-Based Policymaking. September 2017. Accessed October 10, 2018.
About the author:
In 2013, Sherry Glied is Dean of New York University’s Robert F. Wagner Graduate School of Public Service. She held previous appointments as Professor and Chair of Health Policy and Management at Columbia University’s Mailman School of Public Health, Assistant Secretary for Planning and Evaluation at the U.S. Department of Health and Human Services, and Senior Economist for health care and labor market policy on the President’s Council of Economic Advisers under Presidents Bush and Clinton. Dr. Glied recently served as a member of the Commission on Evidence-based Policymaking, which issued its final report, The Promise of Evidence-based Policymaking, to Congress and President Trump in September 2017.
GROWTH: 1000 subscribers and counting!
- September 2018: The Problem with Gun Violence Research: Fear of Uncomfortable Answers~ Author: Alan I. Leshner, MS, PhD
- August 2018: Making Peace with My Friend Jay Dickey~ Author: Mark Rosenberg, MD, MPP
- July 2018: Real-World Evidence to Increase Value in Health Care~Author: McClellan et al.
- June 2018: The Opacity of Transparency: The Search for a Cure to Our Health Care Woes~Author: Niall Brennan, MPP
- May 2018: Why Evidence Matters for Universal Health Coverage: A Perspective from the World Bank Group~Author: Timothy G. Evans, MD, PhD
- April 2018: Contemporary Health Policymaking: Where’s the Evidence?~ Author: Susan Dentzer
- March 2018: Designing a New Medical School for the 21st Century: Kaiser Permanente School of Medicine~Author: Craig W Robbins, MD,MPH
- February 2018: Winds of Change in Publishing~Author: David Sampson, MBA
- January 2018: American Heart Association and American College of Cardiology Guideline Modernization Program/Initiative~Author: Abdul R. Abdullah, M.D.
- October 2017: Understanding Global Health Statistics through Visual Presentations~An Interview with Anna Rosling-Rönnlund, Gapminder Co-Founder and Vice President
- December 2017: Data and Analytics: The Backbone of Global Health~Author: Trevor Mundel, PhD
- November 2017: Resolve: What It Is, Why It’s Needed~Author: Thomas R. Frieden, MD, MPH
- September 2017: Producing guidelines for public health emergencies~Authors: Mauricio Ferri, MD, MS and Susan L Norris, MD, MPH, MS
- August 2017: Measuring the Impact of Public Health Interventions: (Almost) as Important as the Work Itself~Author: Rebecca Martin, PhD
- July 2017: Health Systems – Why do they matter for the developing world?~Author: Kelly Saldaña, MPH
- June 2017: Expanding Transparency in Health Estimation~Author: Christopher J. L. Murray, MD, DPhil
- May 2017: The Long History and Bright Future of Artificial Intelligence in Medicine~Author: Joël Kuiper
- April 2017: A brief overview of development and implementation of clinical practice guidelines in Mexico~Author: José Luis Mayorga Butrón, MD, MSc
- March 2017: Reliable and relevant systematic-reviews/meta-analyses to facilitate guideline production for healthcare: Still room for improvement~Authors: Heidi Tibollo, MLS, and Todd Feinman, MD
- February 2017: ‘Unconference’ to Cap Dual Guideline Events~Author: Peter Wyer, MD
- January 2017: What will 2017 Mean for Evidence~Author: Lisa Simpson, MB, BCh, MPH
- June 2016: Risk of Bias in Randomized Trials~Authors: Gordon H. Guyatt, MD, MSc and Jason W. Busse, DC, PhD
- November 2016: Guidelines and Multimorbidity~Author: Cynthia Boyd, MD, MPH
- October 2016: Advancing quality and science in the practice guideline enterprise using AGREE~Author: Melissa Brouwers, PhD
- September 2016: From Evidence to Decision~Author: Pablo Alonso Coello, MD, PhD
- August 2016: Body of Evidence and GRADE~Author: Philipp Dahm, MD
- July 2016: Gaining confidence in the findings of observational studies~Author: Eddy Lang, MD
- May 2016: Evaluating Systematic Review Quality~Authors: Rebecca L Diekemper, MPH, Belinda K Ireland, MD, MS, and Liana R Merz, PhD, MPH
- April 2016: Assessing the Evidence – Series Launch~Authors: Craig Whittington, PhD & Jacob Franek
- March 2016: Putting the Evidence into AHRQ’s EvidenceNOW~Author: David Meyers, MD
- February 2016: Implementing Person-Centered Care through Guidelines~Author: Anna R Gagliardi, PhD
- October 2015: The Power of Reliable Evidence~Authors: Michael E. Stuart MD, and Sheri Ann Strite
- September 2015: The Institute of Medicine Standards for Systematic Reviews: 4 Years Later~Author: Sally C. Morton, PhD
- August 2015: What is the impact of the IOM Clinical Practice Guidelines Report on consumer stakeholder involvement?~Author: Peg Ford
- July 2015: Effects of the IOM Reports: A Guideline Developer’s Perspective~Author: Tom Oliver
- June 2015: What a difference a year makes: 1 year after implementing the AHRQ National Guidelines~Author: Mary P. Nix, MS, PMP
- May 2015: The Institute of Medicine Standards for Trustworthy Clinical Practice Guidelines: 4 Years Later~Author: Sheldon Greenfield, MD
- March 2015: Initiating a Discussion: Challenges and Opportunities for Use of Real World Data~Author: Sandra Zelman Lewis, PhD
- April 2015: The Guidelines International Network/North America Community: We’ve Come a Long Way~Author: Marguerite Koster, MA, MFT
- February 2015: Orchestrating people, process, and technology in the development and implementation of evidence-based guidelines~Author: Craig W Robbins, MD, MPH
- January 2015: The Evolution of Technology~Author: Craig Whittington, PhD
- November 2014: GROWTH Commentaries Announcement: The Launch of GROWTH Commentaries