The “Science” Behind Generating Headlines

Last December, EPA released a draft report on water quality in tiny Pavillion, Wyo., which was immediately seized upon by opponents of natural gas development in the United States (and even around the world) as smoking-gun proof that hydraulic fracturing pollutes drinking water. Never mind that the paper hadn’t been peer reviewed, or that within a few months the EPA to backtrack and admit that its testing procedures were inadequate, suspending peer review altogether until new sampling could be completed. Just two months after the release of the draft report, EPA Region 8 administrator Jim Martin told a House panel in no uncertain terms that the agency had not established a “causal link” between hydraulic fracturing and water contamination.

Fast-forward to today. Shale opponents have now seized upon yet another “report” (from Cornell, where else?) that supposedly links poor infant health (specifically low birth weight) to natural gas production. And, once again, the paper has not yet undergone peer review — the very process that helps sort out, at least in theory, legitimate scientific conclusions from simple suppositions or even outright activism. In fact, left unmentioned by the activists cheering the release of the paper is the fact that the author, Elaine Hill, is a graduate student in applied economics and management — hardly a field that one would expect to include complex epidemiological assessments.

Andy Revkin at the New York Times – certainly no shill for the oil and gas industry! — has done a deep dive into the problem of jumping the gun on this kind of research, including the fact that opponents are now using Ms. Hill as some sort of “champion” of their cause. What Revkin uncovered, among many things, is that the activist group New Yorkers Against Fracking hired a PR firm (BerlinRosen Public Affairs) to promote the piece, and the firm sent out a pitch to the media about the paper, stating only that it was written by a “researcher at Cornell” — nothing about peer review, and nothing about the fact that the author is still a graduate student. Revkin asked BerlinRosen about why they were promoting a paper before peer review, to which the firm replied that Ms. Hill’s results raise “critical questions” that “should be discussed.”

Why is this worth noting? Because Ms. Hill herself told Mr. Revkin that her results are “preliminary” (that aspect was ignored by activists, either deliberately or inconveniently) and that she “does not want to rush it” in terms of publication. She also said it’s a “valid” point to suggest that her conclusions not be cited until peer review is complete — completely undermining the PR firm (on behalf of New Yorkers Against Fracking) that tried to do exactly the opposite.

EID did its own review of the paper earlier this week, and what follows is a list of some of the most significant concerns and items of interest:

Item I: Skewed results indicate more complexity

Based on Ms. Hill’s results, low birth weight increased by 25 percent for babies born to mothers living within 2.5 km of a well. But at 2.0 km, according to her research, the increase was 26 percent — and at 1.5 km the increase was 21 percent (p. 18). In other words, the closest sample actually had a lower rate of increase than the sample set farther away, with the middle actually recording the highest increase. If there were a “causal relationship” between natural gas development and low birth weight, as she herself claims (see Item II below), why didn’t her results show higher intensity closer to the well? After all, a “causal” relationship means “If A then B,” not “If A then maybe B.”

Item II: Working backwards from a conclusion

The author says up front that her research investigates the “causal relationship between unconventional NGD [natural gas development] and infant health in Pennsylvania” (p. 2). This is not just a strongly worded statement affirming the validity of your hypothesis before peer review (a dangerous game in and of itself), it’s also completely unscientific. When you compile the “first” assessment of anything (as she claims, see p. 4),  you are essentially by any legitimate scientific definition not establishing a “causal” link. But Ms. Hill essentially establishes that link by decree. This could be poor phrasing, but given the conclusions later in the paper that stem from that statement and indeed hinge upon it (more on that below), this seems like more than simply a semantic oversight. (Interestingly, Ms. Hill told Revkin that her language was intentionally strong, but that it wasn’t meant to mislead readers about the “caveats surrounding these findings.”)

Item III: Building toward a convenient narrative

The author couldn’t help but leap from an empirical research project into the policy and advocacy realm, stating at the end of her paper (p. 21-22): “These results suggest that policies that intend to prevent pollution exposure stemming from unconventional natural gas development should increase the regulated/allowable distance between drilling activity and nearby residences.” She then reaches even further, saying that shale development is occurring in 31 states nationwide, which, to her, means that “there is likely to be many exposed babies resulting in a nationwide increase in LBW.” (Interestingly, Ms. Hill has already backtracked from this statement, stating earlier this week that her results actually have “limited external validity” — a complete contradiction of the assertion in her paper.)

Ms. Hill also states that since her paper only looked at the impacts on infant health at birth, “the total increased health costs due to unconventional natural gas development are likely to be much greater.” Again, no credible scientific study would start to make these kinds of extrapolations or recommendations based upon a single assessment that the author herself admits is essentially a first of its kind – unless (a) the purpose was to make those kinds of recommendations and then work backwards from there, or (b) she truly believes that a single initial assessment — limited in scope — scientifically justifies conclusions that require evidence that she has not even collected, much less analyzed.

Item IV: Flawed air pollution claims

Ms. Hill cites air pollution problems in the Dallas-Fort Worth area (p. 9-10), mentioning a single assessment conducted by a consulting group in 2011. What Ms. Hill doesn’t mention is the research that’s been done by the Texas Commission on Environmental Quality (TCEQ), the regulatory body in charge of regulating air emissions in the state, which also has access to the most comprehensive emissions data in Texas. Here’s what TCEQ has said about the region Ms. Hill references:

“After several months of operation, state-of-the-art, 24-hour air monitors in the Barnett Shale area are showing no levels of concern for any chemicals. This reinforces our conclusion that there are no immediate health concerns from air quality in the area, and that when they are properly managed and maintained, oil and gas operations do not cause harmful excess air emissions.”

Ms. Hill also claims there are “no current studies [examining air emissions and toxicities] in the Marcellus shale in Pennsylvania” (p. 10) – a categorically false claim. The Pennsylvania Department of Environmental Protection (DEP) conducted ambient air sampling reports for the Northeast Marcellus and the Southwest Marcellus, both under Gov. Ed Rendell (D). After extensive study, the DEP “did not identify concentrations of any compound that would likely trigger air-related health issues associated with Marcellus Shale drilling activities” in either assessment. Perhaps Ms. Hill was referring to peer-reviewed academic studies, but to pretend these comprehensive regulatory assessments simply don’t exist seems like a huge omission — and perhaps a little too convenient.

Item V: Irrelevant data proves health risks?

The paper cites a paper compiled by Cornell veterinarians (which itself was fundamentally flawed) as essentially scientific proof of her conclusions, even though she admits the lack of applicability: “Although their study is not an epidemiologic analysis, nor is it a study that identifies specific chemical exposures related to NGD, it provides evidence that there are clear health risks in natural gas development” (p. 11). This is an enormous logical fallacy: “A doesn’t provide evidence of B, but A nonetheless clearly proves B.” And remember, a member of the U.N. Environmental Programme called the veterinarians’ paper “an advocacy piece” that “does not qualify as a scientific paper.”

The upshot here is that a paper that has not yet undergone peer review should, as Lisa McKenzie of the Colorado School of Public Health has said, be “approached with extreme caution.” But for opponents of hydraulic fracturing and natural gas development, the need to spread information that maligns the oil and gas industry trumps even basic standards of fact checking, much less a lengthy scientific review process.

This is now the second time in less than a year that anti-shale activists have run with a conclusion that has not been scientifically vetted, and unfortunately for their credibility, it’s also the second time that even a cursory review of available data shows how flawed their conclusions truly were. And that’s in addition to the AP analysis that ran earlier this week, which showed how ideological opponents have seized upon convenient talking points, even when the facts completely contradict them.

Trackbacks

  1. [...] At the time, Energy In Depth highlighted some of the critiques of that study, which was also not peer-reviewed before it was sent to the media, and which the author described as “preliminary.” We haven’t [...]

  2. [...] A similar phenomenon played out in the summer of 2012, when a Cornell graduate student released astudy suggesting a correlation between hydraulic fracturing and low birth weight. The results were blasted out to the media, and several news outlets cheerfully used the study to write another story about the risks of development. Activists from across the country piled on, using the research to make inflammatory claims about how hydraulic fracturing harms children. [...]

Speak Your Mind

*