Measuring public opinion can be a tricky task. The answers you get really depend on who you ask, and how you ask the question, and – less appreciated – where in the interviewing process you ask the question. Ask the wrong group of people a question, or ask the question in the wrong way, or ask the question after a series of other questions, and the results of your public opinion poll won’t accurately measure public opinion.

Unfortunately, a recent Bloomberg National Poll that found an increase in public support for more regulation on hydraulic fracturing appears to have made all three mistakes – asking a question of the wrong group of people, asking it in the wrong way, and asking it after a series of other questions that may have affected the results. As a result, this poll doesn’t add any substance to the debate over hydraulic fracturing, and is actually quite misleading.

The Dec. 13 Bloomberg poll found 66% of the American public wants “more” regulation of hydraulic fracturing – the technology that makes the production of oil and natural gas from deep shale formations possible. That’s an increase of 10 percent since September. There were more than 20 questions in the survey, which took in a wide range of subjects, including the meaning of President Obama’s election win, why Grover Norquist’s Taxpayer Protection Pledge makes Republican lawmakers resistant to raising taxes, and which Democrats would make good presidential candidates in 2016. To conduct the poll, Bloomberg surveyed 1,000 people randomly selected across the country.

Mistake #1 – Asking a national audience about state-by-state issues

It’s understandable that when fielding a national poll, Bloomberg would want to add a question about hydraulic fracturing – in some eastern U.S. states, it’s a topic that’s in the news a good bit. But sampling a group of 1,000 randomly selected adults from all across the country doesn’t provide a meaningful measure of public sentiment toward hydraulic fracturing and the American energy production it makes possible.

In recent years, opponents of the oil and gas industry have intensely lobbied the news media to nationalize the debate over shale development to support a political outcome – the federal government taking regulation of fracturing technology away from the states. It hasn’t worked, and states remain the primary regulators, as they have been since fracturing was first pioneered in the 1940s. As a result, the debate over hydraulic fracturing is also very state-centric, and this issue isn’t covered or talked about equally across the country. In some states, it’s a hot topic, and in others, it’s barely discussed at all. In Florida, few have probably even heard of the thing.

Rather than ask 1,000 randomly selected adults from across the country about hydraulic fracturing, a more relevant exercise would have been polling in states where tight oil and gas development is actually relevant. Had Bloomberg done this, they may have found the public more evenly divided on the question. For example, recent polls from New York have found much narrower margins on questions tied to hydraulic fracturing – 42%-38% support (Siena College Poll), and 44%-42% support (Quinnipiac Poll).

Again, it’s understandable that Bloomberg wanted to write a story about what “the American public” thinks about hydraulic fracturing. But besides generating some (of its own) headlines, polling a national audience fails to provide any meaningful information about how public opinion in individual states may influence regulation in those states. Just as national opinion polls don’t show who’s winning the battleground states during a presidential election, a national poll about hydraulic fracturing tells you nothing about how the issue is being handled state by state.

Mistake #2 – Asking a clearly biased question

The National Council on Public Polls has the following warning for reporters who write about poll numbers:

“You must find out the exact wording of the poll questions. Why? Because the very wording of questions can make major differences in the results. Perhaps the best test of any poll question is your reaction to it. On the face of it, does the question seem fair and unbiased? Does it present a balanced set of choices?”

In other words, how you ask the question can influence the answer you get. Let’s take a look at the question in Bloomberg’s poll:

“A process known as ‘hydraulic fracturing’ or ‘fracking’ involves injecting liquids into the ground. It has resulted in a significant increase in production of natural gas, accompanied by a steep drop in its price. Critics have said it is linked to tainted water supplies and earthquakes. Based just on what you know, do you think there needs to be more regulation or less regulation of fracking?”

This question is hardly neutral. It starts with a poor explanation of hydraulic fracturing, and follows with a description of natural gas production and price trends that’s devoid of any context about job creation and economic growth – topics that most Americans care deeply about. Then the question quotes misinformation from oil and gas critics, and doesn’t even mention what supporters of hydraulic fracturing say. The net result is a question that invites someone to express support for more regulation, because if they don’t, they’re effectively expressing support for “tainted water supplies and earthquakes.”

Bloomberg’s pollsters should rewrite this question to be fair to all sides of the debate. As a starting point, they need look no further than the news copy from the Bloomberg reporter who wrote up the results of the survey:

“Industry groups for chemical, fertilizer and steel companies are trumpeting gains for the U.S. economy as natural gas supplies become more accessible. Low-cost natural gas could generate $72 billion in capital investment as petrochemical companies relocate or boost investments in the U.S., according to the American Chemistry Council.

‘This trend is vital to America’s prosperity,’ Rayola Dougher, chief economist at the American Petroleum Institute, told reporters yesterday. ‘However, costly or duplicative regulation of hydraulic fracturing could be incredibly’ harmful, she said.

And while it’s perfectly reasonable to reference environmental concerns in a question about oil and gas development – or any commercial activity, for that matter – the question shouldn’t simply parrot alarmist and ideologically motivated allegations from anti-industry activists. On “tainted water supplies,” Bloomberg’s pollsters should review what environmental regulators have been saying for years, and the expert conclusion of Stanford University geophysicist Mark Zoback, an advisor to U.S. Secretary Steven Chu on hydraulic fracturing:

“…the injection is typically done at depths of around 6,000 to 7,000 feet and drinking water is usually pumped from shallow aquifers, no more than one or two hundred feet below the surface. Fracturing fluids have not contaminated any water supply and with that much distance to an aquifer, it is very unlikely they could.”

As for earthquakes, Bloomberg’s pollsters should also read the National Research Council’s recent study, which concludes “[t]he process of hydraulic fracturing a well as presently implemented for shale gas recovery does not pose a high risk for inducing felt seismic events.” And they should talk to U.S. Geological Survey geophysicist Bill Ellsworth, who set the record straight after his research was misrepresented by the news media: “We don’t see any connection between fracking and earthquakes of any concern to society.”

Finally, notice that Bloomberg’s question said nothing about the current state of regulation. So when they reference “more regulation or less regulation” … that’s compared to what, exactly? The question provides no information on what regulations are currently in place. For all the respondents know, hydraulic fracturing could be unregulated altogether. This is especially possible among those respondents in areas where “hydraulic fracturing” is rarely if ever discussed (see Mistake #1).

This is actually very important, because despite Bloomberg’s lack of context, the reality is that shale development is heavily regulated. For example, here’s how the U.S. Department of Energy and state-led Ground Water Protection Council summarized it:

“A series of federal laws governs most environmental aspects of shale gas development. For example, the Clean Water Act regulates surface discharges of water associated with shale gas drilling and production, as well as storm water runoff from production sites. The Safe Drinking Water Act regulates the underground injection of fluids from shale gas activities. The Clean Air Act limits air emissions from engines, gas processing equipment, and other sources associated with drilling and production. The National Environmental Policy Act (NEPA) requires that exploration and production on federal lands be thoroughly analyzed for environmental impacts.”

And that doesn’t even include all of the strong state regulations in place, many of which have been updated since the above-referenced report was published.

Would the respondents have given the same answer on “more regulation or less regulation” if they were presented with even a paraphrasing of what’s listed here? How can you ask someone to register an informed opinion about whether hydraulic fracturing needs additional regulation when you don’t even provide basic baseline information?

Mistake #3 – “Priming” the respondents 

When conducting a public opinion poll, there’s another concern about how you ask the question: where it appears in the survey relative to other questions. According to the American Association for Public Opinion Research:

“In some surveys, the order of the questions may be designed to ‘lead’ the respondent to a kind of conclusion that produces a predictable response. This form of bias would not have been present if the prior questions had not been asked. This is also referred to as setting up a ‘context effect.’”

Among pollsters, this is commonly called “priming.” A respondent’s answer to a question can be influenced by the question asked immediately beforehand, because the earlier question “primed” them to think about a particular topic.

In the Bloomberg survey, respondents to the hydraulic fracturing question were primed not just with a question about environmentalism, but a question about one of the biggest environmental issues of the past 20 years:

“Do you believe the temperature of the Earth is or is not warming because of human activity?”

Anyone who has followed the debate over global warming knows that asking someone whether they blame human activity for rising temperatures is just like asking: “Do you care about the environment, or not?” Asking a question like this immediately before the hydraulic fracturing question primes the respondents to give an answer that’s perceived as environmentally conscious. Coupled with the fact that no information on current regulations was provided, it’s easy to see how respondents were primed for a particular answer.

Conclusion

None of this is to suggest that Bloomberg deliberately misled the public. As described above, it’s understandable why a national news outlet would try to gauge national opinions on a subject like hydraulic fracturing. But the combined effect of a biased question and a primed audience clearly steered the respondents towards the “more regulation” answer.