Last September, the Commission on Evidence-Based Policymaking released its report, recommending a series of concrete steps to increase the production and use of evidence in government policymaking. Health care programs account for a very substantial share of government spending, so heightening the use of evidence in the design, implementation, and operation of these programs is of obvious importance.
There is, of course, a long history of bringing evidence to bear in the formulation of health policy. The Office of the Assistant Secretary for Planning and Evaluation (ASPE) at the Department of Health and Human Services (HHS) was established in 1966 with the stated purpose of helping the Department (then Health Education and Welfare) better implement programs. Legislative rules and Office of Management and Budget guidance, such as the requirement that agencies conduct Government Performance and Results Act (GPRA) reviews for ongoing programs and perform Regulatory Impact Analyses (RIAs) for new regulations, formally enshrine a role for evidence in government. But the ecosystem in which government evidence-building and evidence-using activities occur has changed tremendously in the time since ASPE was formed, and the GPRA and RIAs were introduced.
Using evidence to improve government is what taxpayers expect- smart and careful use of limited resources to best address national priorities. -Analytical Perspectives, Budget of the United States Government, Fiscal Year 20181
Take one simple example: back in 1978, a standard government-issue desktop computer had about four kilobytes of core memory and could conduct 300,000 operations in a second. Today, a standard issue cellphone costs about ⅕ as much, has about 800 times as much memory, and conducts about 2000 times as many operations each second. Evidence is produced through the analysis of data. Innovations in computing mean that we could, in principle, have thousands of times more data and analysis available for government decision-making now than we did 40 years ago. Our infrastructure for collecting, analyzing, and using data, however, has not kept pace with this technological progress.
Technological improvements offer new opportunities to turn existing administrative data into a resource that guides decision making. We often think about making better use of data when considering big policy choices, like how to expand health insurance coverage across the US population. Government agencies, including the Congressional Budget Office and ASPE, have often used health data for this purpose. Improvements in computing mean they can do this work better. For example, linking data on individual incomes from the Internal Revenue Service to data on health insurance enrollment decisions could substantially improve our estimates of how people will respond to income-based subsidies.
Data are the lifeblood of decision-making and the raw material for accounatbility. Without high-guality data providing the right infomation on the right things at the right time; designing, monitoring and evalutaing effective policies becomes almost impossible. - United Nations Secretary-General's Independent Expert Advisory Group on a Data Revolution for Sustainable Development2
But some of the biggest technology-driven improvements in the governance of the health care system are likely to come through using our data more routinely, in the context of continuous quality improvements within existing programs. Here, technology offers not only the ability to collect and store data, but the opportunity to use machine learning, A/B testing, low-cost randomized trials, and quasi-experimental methods like regression-discontinuity designs on a routine basis, not just for one-time studies. Imagine if government agencies routinely took advantage of discontinuities generated by scoring algorithms, service interruptions caused by weather disruptions, or alternative formulations of website language to make incremental improvements in the way they operated, just as large technology companies do today. That would require a shift in the evaluation mind-set. Today, agencies formulate questions and then seek data to answer them. In continuous-quality-improvement mode, agencies would regularly look at their data and see what questions they could ask and answer and what improvements they could make with the data they have on hand.
Many of the recommendations of the Commission on Evidence-Based Policymaking would facilitate these changes. One is the recommendation that Congress establish a National Secure Data Service (NSDS). The proposal for development of an NSDS was very much informed by experience with health data, where concerns over both privacy and access have long been important (see Figure 1).
The Commission learned that despite very significant concerns about patient confidentiality, researchers inside and outside government have long had better access to health care data than to many arguably less sensitive government data sources. Paradoxically, one reason for this improved access has been that there is a more robust and well-understood regulatory framework for protecting health data, established through the Health Insurance Portability and Access Act (HIPAA) in 1996. That example taught the Commission that improved privacy and improved access can go hand-in-hand. Promulgating uniform standards limits careless practices that put data at risk and allows establishment of well-understood processes that allow legitimate researchers to use data for statistical purposes (see Figure 2).
Another lesson the Commission learned from the health sphere is that data linkage can be performed effectively without collecting and maintaining a central repository. Storing all federal data in a single monolithic database raises deep and legitimate concerns both about misuse of data by hackers outside government and about government intrusions that limit individual privacy. The NSDS instead is designed as a distributed data model. Researchers can request that a linkage among datasets be made, data can be brought together and combined, and a new linked dataset can be created for statistical analysis. That new dataset can be modified to allow varying levels of individual identification for different releases in more (or less) secure settings and to researchers who are more (or less) trusted. Distributed data resources of this type have been used in a range of healthcare settings, including the Food and Drug Administration’s Sentinel Initiative and the Centers for Medicare and Medicaid Services Edge Server.
The health sector’s experience also informed the Commission’s thinking about how to reorganize government to enhance the routine use of evidence for policymaking and policy implementation (see Figure 3). While ASPE and evaluation offices within the HHS departments have long histories, many government departments and agencies lack a strong evaluation presence. The Commission recommended that every department identify or establish a Chief Evaluation Officer. One role for that officer would be to develop a learning agenda for the Department, identifying key research needs for those within and outside government and helping to develop the evidence base for policy development.
Evidence improves program design and helps existing programs work better. That’s as true in government as in the private sector, where the most successful firms are those that invest in continuous testing of their programs and products. Implementing the recommendations of the Commission on Evidence-Based Policymaking would bring us a big step forward in the development and use of evidence in government. Currently, a bill that would lay the groundwork for implementing the Commission’s recommendations has passed the House and is under consideration in the Senate. But there’s also a lot that agencies like HHS can do today to make progress on improving the use of evidence in its decisions. HHS will soon release a data strategy that articulates priorities for better using their own data, consistent with the commission’s recommendations.
- Office of Management and Budget. Analytical Perspectives, Budget of the U.S. Government, Fiscal Year 2018. Accessed October 10, 2018.
- United Nations, Secretary General’s Independent Expert Advisory Group on a Data Revolution for Sustainable Development. A World That Counts: Mobilizing the Data Revolution for Sustainable Development. New York, NY: United Nations, November 2014. Accessed October 10, 2018.
- Commission on Evidence-Based Policymaking. The Promise of Evidence-Based Policymaking: Report of the Commission on Evidence-Based Policymaking. September 2017. Accessed October 10, 2018.
About the author:
In 2013, Sherry Glied is Dean of New York University’s Robert F. Wagner Graduate School of Public Service. She held previous appointments as Professor and Chair of Health Policy and Management at Columbia University’s Mailman School of Public Health, Assistant Secretary for Planning and Evaluation at the U.S. Department of Health and Human Services, and Senior Economist for health care and labor market policy on the President’s Council of Economic Advisers under Presidents Bush and Clinton. Dr. Glied recently served as a member of the Commission on Evidence-based Policymaking, which issued its final report, The Promise of Evidence-based Policymaking, to Congress and President Trump in September 2017.