A few weeks ago, when researchers at the University of Colorado and the National Oceanic and Atmospheric Administration (NOAA) unveiled their methane emissions study, EID was quick to point out some of its most obvious flaws. Chief among those shortcomings: the fact that it measured methane from only one day of data … from a three hour flight … over only one oil and gas field in the Uintah Basin in Utah … with absolutely no way to determine where the methane is coming from.
Of course, EID wasn’t the only one raising concerns. Energy and climate expert Michael Levi observed that the researchers “only have three hours of observations, and no direct way of knowing whether those observations are representative over longer periods of time.” Steven Hamburg of the Environmental Defense Fund (EDF) urged caution as well, saying that there is “no data to tell us whether the emissions are coming from production, gathering, processing or other activities.”
The researchers themselves even conceded that the limited time window was a significant drawback. As Politico reported: “the authors are already anticipating even more criticism: one criticism [lead researcher Colm Sweeney] acknowledged, was the short duration of the study.”
As it turns out, that was just the tip of the iceberg. As one digs deeper into the research, more and more questions begin to surface.
At the core of the study (and the reason it generated so much press attention) is the suggestion that methane “leakage” during natural gas development is exceedingly high. From the report:
“Given the large global warming potential of CH4, a natural gas leak rate of 6.2 – 11.7 % during production negates any short-term (<70 years) climate benefits of natural gas from this basin for electricity generation compared to oil.”
As noted, NOAA scored some early press coverage on that point back in January with the lead author, Colm Sweeney, proclaiming: “We were expecting to see high methane levels, but I don’t think anybody really comprehended the true magnitude of what we would see.”
This is an interesting comment for two reasons: First, they were expecting to see high methane emissions? Is Sweeney admitting they were going in with preconceived notions? Second, Mr. Sweeney is claiming from one day of data collected in a three hour flight over one oil and gas field that we’re experiencing off the charts methane leakage rates. That’s a pretty serious accusation, and one that would normally call for a great deal more data to corroborate – certainly more than a sampling event that ran about as long as a Notre Dame football game.
Notably, the researchers said that they made twelve flights over Uintah Basin in February 2012, but claimed that eleven of those flights were essentially of no use. As the report puts it:
“Twelve flights were made over the Uintah Basin in February 2012. Non-ideal meteorological conditions (in particular, low, variable, and sometimes recirculating winds in the 0.5 1 1.5 ms range) on the eleven other flight days made direct mass-balance analysis of CH4 emissions impossible.”
But day twelve, February 3, was somehow absolutely perfect:
“Because of the low uncertainty and the fact that the basin was so well cleaned out by the high winds prior to our flight on February 3, the derived emissions estimate from this day is the focus of this study.”
As Sweeney told Politico, the calculations on February 3 were “incredibly accurate and precise.” Based on and compared to what, exactly? The 11 consecutive days of lousy data that preceded it? We can’t know exactly from the information provided. Other than telling us that the other eleven flights were done under “impossible” circumstances, NOAA only gives us this:
“CH4 enhancements measured on the other flight days were large, however, with average mole fractions from 2030 to 2650 ppb inside the PBL (Supplementary Figure 3).”
And yet, the report goes on to acknowledge that the data from all twelve flights were still used by the researchers to draw some of their conclusions, including the claim that emissions are similar every day:
“[There is] no clear evidence in the data from our twelve flight days that a single point source is responsible for a large fraction of the emissions; we infer that it is unlikely that emissions differ drastically from one day to another.”
Measurements from those eleven other “impossible” flights are used to support the idea that the measurements from that perfect twelfth flight represent average conditions in the field. That’s twelve sampling attempts, only one of which was adequate enough to extract data, and the researchers say that single day was perfectly representative of average conditions. Got that?
But wait, it gets even more intriguing:
“The oil and gas wells whose emissions were estimated from our flight transect are almost entirely contained in Uintah County, so we calculate the amount of raw natural gas that would correspond to our estimate CH4 emission and compare it the average hourly natural gas production from Uintah County from both oil and gas wells.”
Emissions measured came from oil and gas wells that were mostly, but not entirely, in Uintah County, yet the researchers compared that measurement against production data from only Uintah County. This means the leakage rate observed is based an improper calculation, to say the least – and one that, by definition, is an inflated figure.
Finally, after the researchers make the bold claim that the Uintah basin has methane emissions rates that are tantamount to a climate crisis, they qualify the statement by noting:
“An inventory analysis by the US Government Accountability Office (GAO) suggests, however, that the fraction of natural gas emissions relative to production from the Uintah, a basin that produces approximately 1% of total US natural gas, is atypical of many Western US basins. Using the Western Regional Air Partnership (WRAP) phase III inventory and production numbers for 2006 from federal leases, the GAO estimates that the production of Uintah natural gas that is flared or vented is much greater (5% of production) than in surrounding regions, including the Denver-Julesburg (2.1%), Piceance (2.5%), N. San Juan (0.34%) and S. San Juan (1.13%) Basins.”
That explains why the researchers were “expecting to see high methane levels,” as the lead author put it: they knew going in that emissions from the region were “atypical.” The GAO report mentioned was released in October 2010, a full 16 months before the CU/NOAA samples were taken. Are the researchers admitting that they specifically chose an area known to have abnormally high emissions? This is particularly troubling, considering the fact that the authors were reaching out to the media about their “findings” long before the study was ever released to the public.
Ironically, this study wasn’t out two weeks before the new EPA Administrator, Gina McCarthy, announced in a speech in Boulder: “Responsible development of natural gas is an important part of our work to curb climate change.” Perhaps Administrator McCarthy was drawing from EPA’s Greenhouse Gas Inventory, which is industry-wide and contains data collected over years, not just one day (or a few hours, for that matter). EPA’s data indicate a leakage rate of just 1.5 percent, which is well below what is needed for natural gas to maintain its climate benefits. And even that level of leakage is likely too high, given how EPA’s data are also an inflation of actual emissions.
It’s also possible that McCarthy was referencing research from MIT, the University of Maryland, multiple reports from the U.S. Department of Energy, and even a Carnegie Mellon study funded in part by the Sierra Club – all of which indicate industry-wide methane leakage rates are far below the catastrophic levels that the CU/NOAA team conveniently leaked to the press back in January.
To conclude: The CU/NOAA study looked at a gas field known to have abnormal emissions levels; took a three-hour sample over an entire region, comparing the data against the number of wells in only a single county; and then suggested that one sampling event after 11 consecutive lousy ones automatically indicated a representative sample of daily emissions.
The researchers do call for “further atmospheric measurements,” and given the numerous shortcomings identified, we don’t think anyone will disagree with that recommendation.