Trust us. We’re experts.
That has been the unstated message for health data scientists for decades. Even though multi-billion-dollar decisions rely on the strength of the evidence being produced in scientific journals on levels and trends in health, scientists have expected those decision-makers to swallow a certain amount of murkiness when it comes to the completeness and quality of the evidence upon which their decisions must be based. That murkiness comes in five main forms:
For the vast majority of health measurement publications in the literature today, the machinery involved in creating estimates remains a mystery. A group of data experts came together a few years ago to try to change that. Scientists and science journal editors from the London School of Hygiene & Tropical Medicine , the World Health Organization , the Institute for Health Metrics & Evaluation (IHME) at the University of Washington, the Bill & Melinda Gates Foundation, the Harvard T.H. Chan School of Public Health, the University of Ottawa, the University of Edinburgh, PLOS Medicine , The Lancet and others created what came to be known as the GATHER guidelines. GATHER stands for Guidelines for Accurate and Transparent Health Estimates Reporting.
The guidelines form a checklist that any researcher can use to make sure that the global health estimates they are producing are making the maximum contribution to the field of health metrics and ultimately to evidence-based decision-making.
Our thinking was quite simple: let’s create a world where “trust us” is no longer acceptable; let’s show what best practice looks like in reporting health estimates. Following the lead of minds like Atul Gawande, who popularized the idea of a checklist for health care, we created a checklist for health metrics. When a new scientific paper is published, there are now 18 items that should be reported alongside that paper.
Working to Build Trust
The thinking was simple, but the work to follow the guidelines is far from easy. Health estimation is a complicated process. Think about what the manual would look like for you to put together a jet airplane by yourself in your backyard (or a lot of people’s backyards in that case). And yet, a manual for creating a jet airplane is a necessary part of the production process. It’s the only way crew after crew after crew can build safe and reliable airplanes.
The Global Burden of Disease (GBD) project is similarly ambitious and complex. As the coordinating center for the GBD enterprise, IHME currently works with 2,434 collaborators in 132 countries to gather, standardize, and analyze data, develop and test various models and techniques, generate estimates using those tools, and then engage the world with those findings. To go from raw input data to findings that are ready to be used in policy takes a full year of work for each annual update of the GBD. And when the GBD Collaborator Network updates those numbers, they update the entire time series, from 1990 to present day (the latest findings will be published in September in The Lancet in a special 20th Anniversary edition of the GBD).
For decision-makers and policy influencers to have faith in these findings, it is important that they undergo a thorough peer review. That process begins when the hundreds of researchers in the Collaborator Network dissect the findings, using their own ground truth as a reference for whatever geography, disease, injury, or risk is their area of expertise. We received more than 20,000 comments from our global collaborators during the latest round of the GBD.
The second phase is the peer review from the journals where the work has been submitted. Because of the complexity of the work, it is critical that reviewers have windows into the work. And once the work is published, the general public needs to be able to look through those same windows.
Last year, the GBD Collaborator Network made the decision to make the GBD compliant with the new GATHER guidelines. They knew this would be a significant amount of work, but the hope is that it will also serve as a strong example. If an enterprise of this scale can follow the checklist, presumably projects with a more narrow scope should be able to comply as well.
One of the most basic guidelines is to be clear about how a set of global health estimates was funded. That should be as simple as reporting the same sort of funding information that most journals now require. But a more difficult requirement is this one:
Provide information on all included data sources and their main characteristics. For each data source used, report reference information or contact name/institution, population represented, data collection method, year(s) of data collection, sex and age range, diagnostic criteria or measurement method, and sample size, as relevant.
Data for the Greater Good
From its inception, IHME has been investing in a data library system that ultimately has proven quite useful in complying with this particular point. We turned that system into the Global Health Data Exchange, the world’s largest catalog of population health data. You can explore it through simple key word searches. Or you can be more specific in asking for the sources to specific aspects of the Global Burden of Disease by using a special tool we created to do just that, the GBD Data Input Sources Tool.
If we are trying to rigorously generate population health estimates, we need to open up the scientific debate.
In that way, we can have faith that we are informing policies in the most meaningful way, that we are providing evidence that can lead to the most impactful therapeutic discoveries, and that we are truly improving the lives of people worldwide.
About the author:
Christopher J.L. Murray, MD, DPhil, is a Professor of Global Health at the University of Washington and Institute Director of the Institute for Health Metrics and Evaluation (IHME) whose career has focused on improving health for everyone worldwide by improving health evidence. A physician and health economist, his work has led to the development of a range of new methods and empirical studies to strengthen health measurement, analyze the performance of public health and medical care systems, and assess the cost-effectiveness of health technologies. IHME provides rigorous and comparable measurement of the world’s most important health problems and evaluates the strategies used to address them.
Dr. Murray is a founder of the Global Burden of Disease (GBD) approach, a systematic effort to quantify the comparative magnitude of health loss due to diseases, injuries, and risk factors by age, sex, and geography over time. After co-authoring the original GBD study in the early 1990s, he brought the GBD enterprise to IHME in 2007 and led the consortium of almost 500 researchers from 50 countries that produced the Global Burden of Diseases, Injuries, and Risk Factors Study 2010 (GBD 2010). GBD now produces annual updates to its estimates and has a current network of 2,434 collaborators in 132 countries. The most recent of the updates, GBD 2015, presents estimates of all-cause mortality, deaths by cause, years of life lost, years lived with disability, and disability-adjusted life years by country, age, and sex from 1990 to 2015.
In his earlier work, Dr. Murray focused on tuberculosis control and the development, with Dr. Alan Lopez, of the GBD methods and applications. As part of this work, they developed a new metric to compare death and disability from various diseases and the contribution of risk factors to the overall burden of disease in developing and developed countries. This pioneering effort continues to be hailed as a major landmark in public health and an important foundation for policy formulation and priority setting.
From 1998 to 2003, Dr. Murray worked at the World Health Organization (WHO), where he served as the Executive Director of the Evidence and Information for Policy Cluster while Dr. Gro Harlem Brundtland was Director-General. He went on to become Director of the Harvard Initiative for Global Health and the Harvard Center for Population and Development Studies, as well as the Richard Saltonstall Professor of Public Policy at the Harvard School of Public Health, from 2003 until 2007.
Dr. Murray has authored or edited 16 books, many book chapters, and more than 350 journal articles in internationally peer-reviewed publications. He and IHME were the subjects of Epic Measures: One Doctor, Seven Billion Patients, which was published by Harper Wave in 2015. He holds Bachelor of Arts and Science degrees from Harvard University, a DPhil in International Health Economics from Oxford University, and a medical degree from Harvard Medical School.