Introduction

Take one simple example: back in 1978, a standard government-issue desktop computer had about four kilobytes of core memory and could conduct 300,000 operations in a second. Today, a standard issue cellphone costs about ⅕ as much, has about 800 times as much memory, and conducts about 2000 times as many operations each second. Evidence is produced through the analysis of data. Innovations in computing mean that we could, in principle, have thousands of times more data and analysis available for government decision-making now than we did 40 years ago. Our infrastructure for collecting, analyzing, and using data, however, has not kept pace with this technological progress.

Technological Improvements

Data are the lifeblood of decision-making and the raw material for accounatbility. Without high-guality data providing the right infomation on the right things at the right time; designing, monitoring and evalutaing effective policies becomes almost impossible. - United Nations Secretary-General's Independent Expert Advisory Group on a Data Revolution for Sustainable Development²

But some of the biggest technology-driven improvements in the governance of the health care system are likely to come through using our data more routinely, in the context of continuous quality improvements within existing programs. Here, technology offers not only the ability to collect and store data, but the opportunity to use machine learning, A/B testing, low-cost randomized trials, and quasi-experimental methods like regression-discontinuity designs on a routine basis, not just for one-time studies. Imagine if government agencies routinely took advantage of discontinuities generated by scoring algorithms, service interruptions caused by weather disruptions, or alternative formulations of website language to make incremental improvements in the way they operated, just as large technology companies do today. That would require a shift in the evaluation mind-set. Today, agencies formulate questions and then seek data to answer them. In continuous-quality-improvement mode, agencies would regularly look at their data and see what questions they could ask and answer and what improvements they could make with the data they have on hand.

Lessons Learned

Many of the recommendations of the Commission on Evidence-Based Policymaking would facilitate these changes. One is the recommendation that Congress establish a National Secure Data Service (NSDS). The proposal for development of an NSDS was very much informed by experience with health data, where concerns over both privacy and access have long been important (see Figure 1).

The Commission learned that despite very significant concerns about patient confidentiality, researchers inside and outside government have long had better access to health care data than to many arguably less sensitive government data sources. Paradoxically, one reason for this improved access has been that there is a more robust and well-understood regulatory framework for protecting health data, established through the Health Insurance Portability and Access Act (HIPAA) in 1996. That example taught the Commission that improved privacy and improved access can go hand-in-hand. Promulgating uniform standards limits careless practices that put data at risk and allows establishment of well-understood processes that allow legitimate researchers to use data for statistical purposes (see Figure 2).

Another lesson the Commission learned from the health sphere is that data linkage can be performed effectively without collecting and maintaining a central repository. Storing all federal data in a single monolithic database raises deep and legitimate concerns both about misuse of data by hackers outside government and about government intrusions that limit individual privacy. The NSDS instead is designed as a distributed data model. Researchers can request that a linkage among datasets be made, data can be brought together and combined, and a new linked dataset can be created for statistical analysis. That new dataset can be modified to allow varying levels of individual identification for different releases in more (or less) secure settings and to researchers who are more (or less) trusted. Distributed data resources of this type have been used in a range of healthcare settings, including the Food and Drug Administration’s Sentinel Initiative and the Centers for Medicare and Medicaid Services Edge Server.

Incorporating Evidence

The health sector’s experience also informed the Commission’s thinking about how to reorganize government to enhance the routine use of evidence for policymaking and policy implementation (see Figure 3). While ASPE and evaluation offices within the HHS departments have long histories, many government departments and agencies lack a strong evaluation presence. The Commission recommended that every department identify or establish a Chief Evaluation Officer. One role for that officer would be to develop a learning agenda for the Department, identifying key research needs for those within and outside government and helping to develop the evidence base for policy development.

Evidence improves program design and helps existing programs work better. That’s as true in government as in the private sector, where the most successful firms are those that invest in continuous testing of their programs and products. Implementing the recommendations of the Commission on Evidence-Based Policymaking would bring us a big step forward in the development and use of evidence in government. Currently, a bill that would lay the groundwork for implementing the Commission’s recommendations has passed the House and is under consideration in the Senate. But there’s also a lot that agencies like HHS can do today to make progress on improving the use of evidence in its decisions. HHS will soon release a data strategy that articulates priorities for better using their own data, consistent with the commission’s recommendations.