Evaluating the Expertise of Climate Scientists

James Hansen took his degree in astrophysics, not normally a climate-related field. Nonetheless, few would argue that he is not an expert on climate change.

On the other hand, Freeman Dyson is possibly the second smartest person on the planet, a theoretical physicist who worked in the field of climate science for 15 years. And yet, because he does not support the consensus, climate activists dismiss him as unqualified.

How do we estimate the expertise of someone in a field where we ourselves are not expert?

This is a current events question, given the recent publication of ‘Consensus on Consensus: A Synthesis of Consensus Estimates on Human-Caused Global Warming,’ written by (among others) John Cook, Naomi Oreskes, Stefan Lewandowsky and William Anderegg, all authors of papers much criticized here.

The point of their paper is simple: The work of some of the co-authors of the paper were cricitized by Richard Tol. The thrust of his criticism is that many studies of climate consensus eliminate large amounts of data considered unqualified by the researchers. Tol writes, “Cook et al (2013) estimate the fraction of published papers that argue, explicitly or implicitly, that most of the recent global warming is human-made. They find a consensus rate of 96%–98%. Other studies6 find different numbers, ranging from 47% in Bray and von Storch (2007) to 100% in Oreskes (2004)—if papers or experts that do not take a position are excluded, as in Cook et al.If included, Cook et al find a consensus rate of 33%–63%. Other studies range from 40% in Bray and von Storch (2007) to 96% in (Carlton et al 2015). Cook et al use the whole sample. Other studies find substantial variation between subsamples. Doran and Zimmerman (2009), for instance, find 82% for the whole sample, while the consensus in subsamples ranges from 47% to 97%. Verheggen et al (2014) find 66% for the whole sample, with subsample consensus ranging from 7% to 79%.”

This most recent paper by those for whom the Tol belled is an attempt to justify their decisions. Their reasoning is simple. If you eliminate the non-experts from the total being surveyed, the experts will agree with you.

In the Supplementary Information to their paper they write, “We define domain experts as scientists who have published peer-reviewed research in that domain, in this case, climate science.” (Despite this, they eliminate many peer-reviewed respondents in Verheggen et al, for example.)

As I mentioned the other day, a simple publication count is a remarkably weak way of estimating expertise. I wrote, “The weaknesses of publication records are:

1. Very capable younger scientists have not had time to establish a record of publications. Dismissing their opinions leads to loss of useful information.

2. As ‘alarmists’ like to point out whenever an older scientist expresses a skeptical viewpoint, at some point in the natural cycle of a person’s career, ongoing education becomes less important. One can make the case that someone reaching the end of their career actually knows less than a freshly minted scientist.

3. The tools and techniques used in tertiary education are different than they were when many older scientists were educated. In addition, new knowledge is incorporated into texts available to younger scientists. This again may advantage the young at the expense of the old.

4. Some scientists are co-authors of numerous papers for reasons other than their ability to contribute to the main body of the scientific arguments advanced in the paper. Their publication count may be more impressive than their actual command of the field.

5. Some very good scientists work outside the academic world and publication may not be a priority for them. Using publications as a proxy for expertise again may devalue their opinions.”

When I made those points to another of the paper’s co-authors (Bart Verheggen), he agreed but basically said it was the only way he could think of.

While Lewandowsky, Cook and Oreskes are not climate scientists, it seems that none of the team involved with the paper thought to look at how others evaluate expertise. It didn’t occur to them that there is a body of work that could have informed their paper. As many of the co-authors were in fact authors of papers cited in the most recent work, it really seems as if they missed the boat.

This surprises me a little, given the frequency with which they throw around the term ‘Dunning Kruger Effect,’ which describes the tendency of individuals to overestimate their own knowledge or abilities. It’s part of the field of expertise evaluation, yet that name is the only thing that seems to have stuck.

Expert recommendations are an oft-used technique to identify those with expertise. People refer those they think are experts and if enough of them do it,they are awarded the title.

Expertise is a highly relevant topic in the field of law, where my expert goes against your expert in the courtroom, and establishing who’s better is pretty important. It’s also important in discovery, especially with the new game of ‘e-discovery’, the evaluation of mountains of documents using software to sort it. Again, this is a well-researched topic ignored by Cook et al.

It’s relevant to military decision making, high technology research, and in academia.

In academia,a publication count is considered the crudest method of evaluating expertise, mostly for the reasons I cited above. More common are techniques such as citation measuring (how many times your work has been referenced by others) or impact measurement (the perceived quality of the journals where you are published, often combined with publication counts and citation counts).

I mentioned in my previous post that few of the co-authors have expertise in climate science. Fewer have experience in surveys. None appear to have relevant expertise in evaluating expertise.

They did not utilize the methods most commonly used and most trusted in academia. Worse, they do not appear to have consulted the large body of literature on the subject. There are no references to the appropriate literature in their paper.

They…just did a pub count and called it a day.

There’s no doubt they desperately need to defend their 97% claim of consensus. Going after Exxon and threatening to put skeptics in jail requires that high a level of confidence.

But you would think that if they were going to defend it, they would do a better job.

Most of the original papers referenced in their latest effort are remarkably weak. It seems they didn’t learn from experience.




18 responses to “Evaluating the Expertise of Climate Scientists

  1. Apparently these climate scientists grossly overestimated their expertise in setting up and evaluating scientific questionnaires. Typical Dunning-Kruger.

  2. I’ve seen climate papers with 20-30 authors. And they are just 8-10 pages long. What effect does that have?

  3. What is apparently one of their main tools is to filter publications for key words and then to declare the paper supports their bizarre assertion. Some of the papers they refer to have nothing to even partially support their cynical effort.

  4. This whole thing is so far into the trap of the information deficit model. With all do respect to Bart. his paper – and your response to it – asks and answers the wrong question.

    Think of how better it would be were all the time, money and effort spent on studying the consensus of the consensus of the consensus spent, instead, on the actual deployment of emission reducing technologies and efficiencies.

    • Since the benefit of lower emissions on the cliamte is dubious at best, how about we spend the money and effort on how to make the available energy sources as clean, reliable and low cast as possible in the context of actual performance. Not based on fantasy models and useless projections. That cuts out solar and wind completely.

    • Nuclear molten salt is the only no regret option. And yes, China and India are the only countries that really matter in climate policy.

  5. To combat the coming Zika epidemic, we must enlist the expertise of oncologists, podiatrists, osteopaths, and proctologists.

    • no true Scotsman fallacy.

      • It speaks to expertise. To hold an expert opinion on “The consensus” requires that one have expertise in climate sensitivity and attribution, not in paleo-climatology or glaciology. It would be like going to a proctologist to diagnose cancer… On the other hand, a “non-expert”, who experienced the same cancer and had studied up on it, would be far more knowledgeable on the subject than your typical podiatrist.

        Just because someone can be classified as a “climate scientist” does not suggest that they are knowledgeable about specific questions and “what percentage of climatic warming can be directly attributable to human activity” is a very, very specific question.

      • The thing is that climate science is so polarised that even experts like Lewis are a priori dismissed as pranks.

  6. Tom, I just read this link from Judith Curry’s site. It’s a very interesting read.https://c.ymcdn.com/sites/www.ospe.on.ca/resource/resmgr/DOC_advocacy/2015_Presentation_Elec_Dilem.pdf

    This slide says it all.

    Why Will Emissions Double as We Add Wind and Solar Plants?

    ² Wind and Solar require flexible backup genera,on.
    ² Nuclear is too inflexible to backup renewables without expensive engineering changes to the reactors.
    ² Flexible electric storage is too expensive at the moment.
    ² Consequently natural gas provides the backup for wind and solar in North America.
    ² When you add wind and solar you are actually forced to reduce nuclear generation to make room for more natural gas genera,on to provide flexible backup.
    ² Ontario currently produces electricity at less than 40 grams of CO2 emissions/kWh.
    ² Wind and solar with natural gas backup produces electricity at about 200 grams of
    CO2 emissions/kWh. Therefore adding wind and solar to Ontario’s grid drives CO2 emissions higher. From 2016 to 2032 as Ontario phases out nuclear capacity to make room for wind and solar, CO2 emissions will double (2013 LTEP data).
    ² In Ontario, with limited economic hydro and expensive storage, it is mathematically impossible to achieve low CO2 emissions at reasonable electricity prices without nuclear generation.

    • Almost Iowa,

      No, the slide does not “say it all”. It is specific to one unusual place occupied by 0.2% of the planet’s population.

      Slide 18 from the same presentation makes a much more general point: “We need to stop adding solar and wind for ideological reasons and focus on using them only when their economic and environmental contributions are positive.” Also, see the summary slide.

      Texas now gets more than 10% of its electricity from wind, the amount is still increasing. And now a lot of solar is coming on line there. But that is a place where wind and solar make a lot more sense than Ontario or Germany.

      • “”No, the slide does not “say it all”. It is specific to one unusual place occupied by 0.2% of the planet’s population.”

        Actually – no. It is relevant to any place where wind and solar displace nuclear.

        Let’s take the case of wind. The ONLY reason to build costly, inefficient and ineffective wind farms is to cut emissions, however, since wind turbines only produce at 30% of capacity, it necessitates running back up gas fired plants to meet demand when the wind blows too hard or not. This is what they mean by:

        “Wind and solar with natural gas backup produces electricity at about 200 grams of CO2 emissions/kWh. Therefore adding wind and solar to Ontario’s grid drives CO2 emissions higher. From 2016 to 2032 as Ontario phases out nuclear capacity to make room for wind and solar, CO2 emissions will double (2013 LTEP data).”

        We face the exact same situation in Minnesota and Germany suffers from the same problem. It is more than just Ontario.

        “Texas now gets more than 10% of its electricity from wind, the amount is still increasing.”

        Double check your numbers. I would agree that 10% of the electrical producing capacity in Texas is 10% but that is a far cry from “gets more than 10% of its electricity from wind”. Typically, one can expect only 30% of capacity from wind.

  7. Almost Iowa,

    You wrote: “Actually – no. It is relevant to any place where wind and solar displace nuclear.”

    How many places is that? Ontario, France, maybe Germany if they keep pushing wind and solar to an insane degree. Throw in displacing hydro, as we should, and you can add Quebec, the Pacific Northwest, Scandinavia. I may well be missing some other places that get virtually all baseload from nuclear and/or hydro, but they are very few. The norm is fossil fuel.

    You wrote: “Double check your numbers. … Typically, one can expect only 30% of capacity from wind.”

    You are the one who needs to double check. I based my statement on 2013, when ERCOT (serving about 85% of Texas) had about 11 GW of wind capacity producing an average of 3.7 GW of power out of a total of 37 GW. According to Wikipedia (https://en.wikipedia.org/wiki/Wind_power_in_Texas#Statistics) Texas now has 17.1 GW of wind capacity and got an average of 5.1 GW production in 2015. So approaching 15% of total production for ERCOT, over 12% for Texas as a whole.

    They seem to have a whole lot more wind and solar in the pipeline. Since that wind expansion started, electricity prices have gone down. That is due to the drop in natural gas prices, but obviously all that wind has not broken the bank.

    • Almost Iowa,

      You wrote: “We face the exact same situation in Minnesota”.
      I decided to double check your numbers. Minnesota gets just 13% of its power from nuclear and emits 570 kg CO2 per MWh generated, 14 times as much as Ontario. Wind will displace fossil fuel, not nuclear.

      Wind is not really displacing nuclear in Germany since their idiotic decision to phase out nuclear was driven by other stupid political reasons.

  8. Reblogged this on 4timesayear's Blog.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s