Trust us. We’re experts.

That has been the unstated message for health data scientists for decades. Even though multi-billion-dollar decisions rely on the strength of the evidence being produced in scientific journals on levels and trends in health, scientists have expected those decision-makers to swallow a certain amount of murkiness when it comes to the completeness and quality of the evidence upon which their decisions must be based. That murkiness comes in five main forms:

For the vast majority of health measurement publications in the literature today, the machinery involved in creating estimates remains a mystery. A group of data experts came together a few years ago to try to change that. Scientists and science journal editors from the London School of Hygiene & Tropical Medicine , the World Health Organization , the Institute for Health Metrics & Evaluation (IHME) at the University of Washington, the Bill & Melinda Gates Foundation, the Harvard T.H. Chan School of Public Health, the University of Ottawa, the University of Edinburgh, PLOS Medicine , The Lancet and others created what came to be known as the GATHER guidelines. GATHER stands for Guidelines for Accurate and Transparent Health Estimates Reporting.

The guidelines form a checklist that any researcher can use to make sure that the global health estimates they are producing are making the maximum contribution to the field of health metrics and ultimately to evidence-based decision-making.

Our thinking was quite simple: let’s create a world where “trust us” is no longer acceptable; let’s show what best practice looks like in reporting health estimates. Following the lead of minds like Atul Gawande, who popularized the idea of a checklist for health care, we created a checklist for health metrics. When a new scientific paper is published, there are now 18 items that should be reported alongside that paper.

Working to Build Trust

The thinking was simple, but the work to follow the guidelines is far from easy. Health estimation is a complicated process. Think about what the manual would look like for you to put together a jet airplane by yourself in your backyard (or a lot of people’s backyards in that case). And yet, a manual for creating a jet airplane is a necessary part of the production process. It’s the only way crew after crew after crew can build safe and reliable airplanes.

The Global Burden of Disease (GBD) project is similarly ambitious and complex. As the coordinating center for the GBD enterprise, IHME currently works with 2,434 collaborators in 132 countries to gather, standardize, and analyze data, develop and test various models and techniques, generate estimates using those tools, and then engage the world with those findings. To go from raw input data to findings that are ready to be used in policy takes a full year of work for each annual update of the GBD. And when the GBD Collaborator Network updates those numbers, they update the entire time series, from 1990 to present day (the latest findings will be published in September in The Lancet in a special 20^th Anniversary edition of the GBD).

For decision-makers and policy influencers to have faith in these findings, it is important that they undergo a thorough peer review. That process begins when the hundreds of researchers in the Collaborator Network dissect the findings, using their own ground truth as a reference for whatever geography, disease, injury, or risk is their area of expertise. We received more than 20,000 comments from our global collaborators during the latest round of the GBD.

The second phase is the peer review from the journals where the work has been submitted. Because of the complexity of the work, it is critical that reviewers have windows into the work. And once the work is published, the general public needs to be able to look through those same windows.

Last year, the GBD Collaborator Network made the decision to make the GBD compliant with the new GATHER guidelines. They knew this would be a significant amount of work, but the hope is that it will also serve as a strong example. If an enterprise of this scale can follow the checklist, presumably projects with a more narrow scope should be able to comply as well.

One of the most basic guidelines is to be clear about how a set of global health estimates was funded. That should be as simple as reporting the same sort of funding information that most journals now require. But a more difficult requirement is this one:

Provide information on all included data sources and their main characteristics. For each data source used, report reference information or contact name/institution, population represented, data collection method, year(s) of data collection, sex and age range, diagnostic criteria or measurement method, and sample size, as relevant.

Data for the Greater Good

From its inception, IHME has been investing in a data library system that ultimately has proven quite useful in complying with this particular point. We turned that system into the Global Health Data Exchange, the world’s largest catalog of population health data. You can explore it through simple key word searches. Or you can be more specific in asking for the sources to specific aspects of the Global Burden of Disease by using a special tool we created to do just that, the GBD Data Input Sources Tool.

If we are trying to rigorously generate population health estimates, we need to open up the scientific debate.

In that way, we can have faith that we are informing policies in the most meaningful way, that we are providing evidence that can lead to the most impactful therapeutic discoveries, and that we are truly improving the lives of people worldwide.