Rick Santorum, Politifact, Bart Verheggen, Fabius Maximus and the problem with surveys

If someone ever asks you to write a report of the results of a survey, remember one thing.

If you want to be taken seriously, shine a light on your problems. If you want the answer to be X and the survey result is Y, talk about it first. Talk about it often. That way nobody can say ‘Gotcha!’

Bart Verheggen conducted a survey of over 1,800 scientists on their belief in (and confidence in that belief) the amount of human contributions to recent climate change.

They reported on it here and here.

I criticized the report (not the survey, which I think is very good) here and here and here.

Fabius Maximus then reinterpreted the results here.

Rick Santorum then used Fabius Maximus’ interpretation in an interview with Bill Maher here:

Politifact then critiqued Santorum’s claims here. Bart Verheggen also critiqued Santorum here.

Everybody got it wrong. Politifact got numbers wrong, confused surveys and didn’t report accurately on what they found.

Santorum got it wrong because Fabius Maximus got it wrong. FM mistakenly inferred beliefs on the part of respondents that led him to understate the strength of the consensus. Santorum copied his mistake in his interview.

Verheggen et al got it wrong. They mistakenly inferred beliefs on the part of respondents that led them to overstate the strength of the consensus. They also committed the first cardinal sin of report writing–they tried to bury the bad news.

They didn’t refer to the headline finding by name. They tried to combine the responses to the headline question with another question and referred to it only by question number.

They didn’t want to report that ‘only’ 66% of the respondents agree that half or more of current warming is caused by human emissions of greenhouse gases. (Wording of the question meant that some scientists could agree with the premise but not be willing to put a specific percentage on human contributions.)

I do this for a living. I make mistakes and I’m not perfect, but I consider myself a subject matter expert on the details of collecting data from surveys and reporting it accurately. (I would rather have been a philosopher or poet, but…)

Here is what I wrote on Bart Verheggen’s weblog today. I stand by it as the most accurate reporting I have seen yet on this survey:

“Your survey is both good and useful. As I tried to warn you in advance, your reporting has left the door open for deliberate misinterpretation.

There is a strong consensus among climate scientists that human emissions of CO2 have caused half or more of recent warming.

66% specifically attributed half or more to emissions. It is possible that more may have done so had the questions been worded differently.

Those scientists with more numerous publications were more likely to make that attribution and were more confident in their judgment.

However, it is clear that the consensus among climate scientists in this survey does not approach the near unanimous agreement found in recent literature searches (Oreskes, Anderegg, Prall et al, Cook et al). Instead, the broad category of climate scientists in agreement on human contributions matches almost exactly that found in Bray, von Storch et al.

The findings of your paper suggest that more research is appropriate, both in attribution and confidence, but also in discovering why scientists with fewer publications do not share the same confidence as their colleagues with more publications.”

Tom_Scott_stupid_fight-570x384

5 responses to “Rick Santorum, Politifact, Bart Verheggen, Fabius Maximus and the problem with surveys

  1. Consider the issue of overconfidence:
    https://en.wikipedia.org/wiki/Overconfidence_effect

    The most common way in which overconfidence has been studied is by asking people how confident they are of specific beliefs they hold or answers they provide. The data show that confidence systematically exceeds accuracy, implying people are more sure that they are correct than they deserve to be. If human confidence had perfect calibration, judgments with 100% confidence would be correct 100% of the time, 90% confidence correct 90% of the time, and so on for the other levels of confidence. By contrast, the key finding is that confidence exceeds accuracy so long as the subject is answering hard questions about an unfamiliar topic. For example, in a spelling task, subjects were correct about 80% of the time, whereas they claimed to be 100% certain.[3] Put another way, the error rate was 20% when subjects expected it to be 0%. In a series where subjects made true-or-false responses to general knowledge statements, they were overconfident at all levels. When they were 100% certain of their answer to a question, they were wrong 20% of the time.[4]

  2. I used to have access to a data base which held confidence levels documented by geoscientists, used to judge oil exploration prospects. In general they were over confident. But the less confident they were, the more overconfident they turned out to be. Thus if they expressed 20 % confidence they were usually wrong. If they expressed 80 % confidence the results were close. There were useless efforts to show them results, but it never seemed to sink in. So we started tossing out anything with low confidence, because the results were garbage.

    I think the best way to ask this may be: are you highly confident that anthropogenic factors drove a) zero to 20 % b) 20 to 40 % c) 40 to 60 % ……etc of the temperature increase observed in the last 50 years? Just have them punch a, b, c, d, e, f, if they are highly confident. Have a g answer: I’m not highly confident.

  3. 66% really is a slam dunk. Cost Benefit of economists show an impetus for policy even if warming is only 10% due to GHG. Perhaps the better question is what percent believe GHG’s are a driver at that level. Give it up. The scientists won.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s