There are many published studies claiming air quality causes adverse health effects, including mortality. But a sticky question remains – does air quality cause adverse health effects, or is the effect an artifact of selective analysis methods and choice of data? The heart of the problem is the analysis of large, complex observational data sets. Analysis flexibility allows researchers to declare statistical significance by how they do the analysis. A chance, positive effect can lead to publication. Epidemiology studies showing that air quality is associated with mortality seem a fluke, given the many possible ways in which the analysis can be done, and unaccounted confounding. The existence of negative studies argues against real effects. Other claims in epidemiology literature may fit this situation.

There are literally thousands of epidemiology (observational) studies in the literature claiming that air quality causes adverse health effects, including mortality. There are three profound examples where it has caused mortality. The best known is the London Fog of November 1952.¹ A temperature inversion trapped smoke, soot, and hazardous gases for five days. Streetlights had to be turned on during the day. Analysis indicated that about 12,000 excess deaths occurred during that period and over the next several months. The other two examples—with similar conditions—occurred in Meuse River Valley, Belgium (1930) and Danora, Pennsylvania (1948).² These unique conditions do not exist today because of clean air regulations and source emission controls.

Americans spend heavily each year to improve air quality. As just one example, a 2011 US Environmental Protection Agency (EPA) study estimated the costs of compliance for regulations under 1990 amendments to the Clean Air Act in the US at approximately $65 billion annually.³ The study also estimated that the regulations would prevent 230,000 cases of premature mortality in 2020. But do these regulations prevent premature mortality, and are epidemiology claims that current air quality causes adverse health effects the result of a fluke of statistical analysis?

With the interest by the EPA (and the public) in understanding what may be the harmful component in the air, two epidemiology studies were published in the 1990s, both partially funded by the EPA. One was the 1993 Harvard Six Cities study of 8,111 adults aged 25‒74 followed for 16 years in six American cities, over which time 1,430 deaths occurred.⁴ The other was a 1995 National Institute of Statistical Science (NISS) study of over 785,000 deaths among adults ≥65 years old in two American counties over a six-year period.⁵ The Harvard study claimed that fine particles in air (PM2.5) contributed to excess mortality, increasing the death rate by 26%. The NISS study used daily deaths and found no association of PM2.5 with mortality.

There were consternation and debate about the Harvard claim, so much so, that the EPA secured the data set and, along with industry, funded other experts to reanalyze it in 2000.⁶ This study found that if one closely followed the Harvard analysis steps, they got much the same result. The effect, however, fell apart if they varied the analysis method. The EPA did not ask NISS for their data set to have it reanalyzed. There have been many air quality‒health effect studies published since. But a sticky question remains – does air quality cause adverse health effects, including mortality or is the effect a methodological (or data) artifact?

The heart of the problem is the analysis of large, complex observational data sets. Some early insight into this problem is helpful. Researchers at Yale in 1988 identified claims for 56 different medical and pharmaceutical exposure topics from published observational studies.^7,8 They then looked at other similar published studies for the same topics and same period and found 136 studies supporting the claims and 127 studies contradicting the claims. In another situation, the president of the Biometrics Society in his 2002 Presidential Address noted that when university students in his regression class were given the same data set and the same question, they came up with dramatically different answers.⁹

Recent research bears these findings out. Multiverse analysis (many analyst studies) using Specification Curve Analysis, SCA, finds that with large, complex data sets, there can be thousands of ways to run an analysis, all seemingly reasonable.^10,11 Here SCA researchers gave the same data set and posed the same question to independent teams. One might think these analyses would converge to the same answer. But these teams consistently produced dramatically different answers. It is starting to look like the analysis method is the fluke causing the research claim.

As a specific example, Smith¹² illustrated how different statistical modeling assumptions for short-term PM2.5‒mortality associations in elderly people gives different results. Smith analyzed data from the Center for Medicare and Medicaid Services, the EPA, and the National Oceanic and Atmospheric Administration. He showed that by simply removing one covariate from the analysis—previous day's temperature—PM2.5‒mortality associations went from being not significant to statistically significant.

Going back to the Harvard‒NISS findings, two later studies were published—in 2011 and 2017—using larger data sets than Harvard, and different methods.^13,14 The 2011 data set included individual-level information on time of death and age for 3.2 million deaths in a population of 18.2 M people at 814 locations across the US throughout the period 2000–2006. The researchers estimated effects across and within locations. They found effects across locations but noted that these effects were likely due to unmeasured (unaccounted) confounders. As to within locations, over time analysis, they noted “we are not able to demonstrate any change in life expectancy for a reduction in PM2.5.”

The 2017 data set included individual-level information on time of death and age for over 2 million deaths throughout the period 2000–2012. One million deaths from ~20M people in eight populous air basins in California were used in the analysis. Within the eight locations, with over 37 thousand exposure days, these researchers found no association of PM2.5 and daily deaths. Also, current estimates of the air quality‒mortality risk are very small, a risk ratio of 1.0065.¹⁵

The world does not see all of what is happening in science. The Harvard study was small (population 8,111, 1,430 deaths) with a positive effect. The effective sample size was six as the statistical comparisons were across cities. The 2011 and 2017 studies were larger: population 18.2M (3.2M deaths) and population ~20M (~1M deaths), and both with no detected effects. Given that many negative studies go unpublished and are put in the file drawer,¹⁶ researcher bias can nudge non-significant results to significant results in other studies,¹⁷ and publication bias of journals to only publish results that are statistically or clinically significant (i.e., positive studies),¹⁸ false positive studies can end up canonized in the literature.¹⁹ This is not unusual. Academics make up most of the journal peer review and editorial positions²⁰ and the academic publishing and tenure process favors the natural selection of bad science.²¹

The Harvard study method comes across as a fluke given the many possible ways in which analysis of large, complex data sets can be done,^10,11 unaccounted confounding (i.e., omitted or missing variable bias),²² and the existence of larger negative studies.^5,13,14 A National Association of Scholars report offers further insights into selective methods that lead to irreproducible research claims for PM2.5 and mortality, heart attacks, and asthma.²³

Air quality and mortality have been used here as an example of how an outcome from a fluke statistical analysis can become an established narrative. But there are likely other exposure topics in epidemiology literature that fit this situation.

1.

Bell, ML, Davis, DL. Reassessment of the lethal London Fog of 1952: Novel indicators of acute and chronic consequences of acute exposure to air pollution. Environmental Health Perspectives, 109(Suppl. 3):389‒94, 2001. JSTOR, https://doi.org/10.2307/3434786

2.

Young, SS, & Kindzierski, WB. Air Quality and Public Health: Is There a Link? Washington, DC: The Heritage Foundation, 2024. https://www.heritage.org/climate/report/air-quality-and-public-health-there-link

3.

US EPA. Benefits and Costs of the Clean Air Act 1990-2020. Report Documents and Graphics, 2011. https://www.epa.gov/clean-air-act-overview/benefits-and-costs-clean-air-act-1990-2020-report-documents-and-graphics

4.

Dockery, DW, Pope III, CA, Xu, X, Spengler, JD, Ware, JH, Fay, ME, Ferris, BG, Speizer, FE. An association between air pollution and mortality in six U.S. cities. New England Journal of Medicine, 329:1753–59, 1993. https://doi.org/10.1056/nejm199312093292401

5.

Styer, P, McMillan, N, Gao, F, Davis, J, Sacks, J. Effect of outdoor airborne particulate matter on daily death counts. Environmental Health Perspectives, 103, 5:490–97, 1995. https://doi.org/10.1289/ehp.95103490

6.

Krewski, D, Burnett, RT, Goldberg, MS, Hoover, K, Siemiatycki, J, Jerrett, M, Abrahamowicz, M, White, W. Reanalysis of the Harvard Six Cities Study and the American Cancer Society Study of Particulate Air Pollution and Mortality. Special Report. Cambridge, MA: Health Effects Institute, 2000. https://www.healtheffects.org/publication/reanalysis-harvard-six-cities-study-and-american-cancer-society-study-particulate-air

7.

Mayes, LC, Horwitz, RI, Feinstein, AR. A collection of 56 topics with contradictory results in case-control research. International Journal of Epidemiology, 17(3):680–85, 1988. https://doi.org/10.1093/ije/17.3.680

8.

Feinstein, AR. Scientific standards in epidemiologic studies of the menace of daily life. Science, 242:1257–63, 1988. https://doi.org/10.1126/science.3057627

9.

Breslow, NE. Are statistical contributions to medicine undervalued? Biometrics, 59:1–8, 2003. https://doi.org/10.1111/1541-0420.00001

10.

Schweinsberg, M, Feldman, M, Staub, N, van den Akker, OR, van Aert, RC, Van Assen, MA, et al. Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes, 165:228–49, 2021. https://doi.org/10.1016/j.obhdp.2021.02.003

11.

Breznau, N, Rinke, EM, Wuttke, A, Nguyen, HH, Adem, M, Adriaans, J, et al. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences, 119(44):e2203150119, 2022. https://doi.org/10.1073/pnas.2203150119

12.

Smith, RL. Dependence of Short-Term Mortality on Fine Particulate Matter in the Population of Elderly Medicare Beneficiaries. November 2021. https://rls.sites.oasis.unc.edu/postscript/rs/Smith-Medicare-PM.pdf

13.

Greven, S, Dominici, F, Zeger, S. An approach to the estimation of chronic air pollution effects using spatio-temporal information. Journal of the American Statistical Association, 106(494):396–406, 2011. https://doi.org/10.1198/jasa.2011.ap09392

14.

Young, SS, Smith, RL, Lopiano, KK. Air quality and acute deaths in California, 2000–2012. Regulatory Toxicology and Pharmacology, 88:173–84, 2017. https://doi.org/10.1016/j.yrtph.2017.06.003.

15.

Orellano, P, Reynoso, J, Quaranta, N, Bardach, A, Ciapponi, A. Short-term exposure to particulate matter (PM10 and PM2.5), nitrogen dioxide (NO2), and ozone (O3) and all-cause and cause-specific mortality: Systematic review and meta-analysis. Environment International, 142:105676, 2020. https://doi.org/10.1016/j.envint.2020.105876

16.

Simonsohn, U, Nelson, LD, Simmons, JP. P-curve: a key to the file-drawer. Journal of Experimental Psychology: General, 143(2):534–47, 2014. https://psycnet.apa.org/record/2013-25331-001

17.

Ioannidis, JP. Why most published research findings are false. PLoS Medicine, 2(8), e124, 2005. https://doi.org/10.1371/journal.pmed.0020124

18.

Dalton, JE, Bolen, SD, Mascha, EJ. Publication bias: The elephant in the review. Anesthesia & Analgesia, 123(4), 812–13, 2016. https://doi.org/10.1213/ANE.0000000000001596

19.

Nissen, SB, Magidson, T, Gross, K, Bergstrom, CT. Publication bias and the canonization of false facts. eLife, 5:e21451, 2016. https://doi.org/10.7554/elife.21451

20.

el-Guebaly, N, Foster, J, Bahji, A, Hellman, M. The critical role of peer reviewers: Challenges and future steps. Nordic Studies on Alcohol and Drugs, 40(1), 14‒21, 2022. https://doi.org/10.1177/14550725221092862

21.

Smaldino, PE, McElreath, R. The natural selection of bad science. Royal Society Open Science, 3(9):160384, 2016. https://doi.org/10.1098/rsos.160384

22.

Bruns SB, Ioannidis JP. p-Curve and p-hacking in observational research. PLoS One, 11(2):e0149144, 2016. https://doi.org/10.1371/journal.pone.0149144

23.

Young, SS, Kindzierski, W, Randall, D. Shifting Sands: Unsound Science and Unsafe Regulation, Keeping Count of Government Science: P-Value Plotting, P-Hacking, and PM2.5 Regulation. New York, NY: National Association of Scholars, 2021. https://www.nas.org/reports/shifting-sands-report-i