Friday, May 17, 2013

Cook's survey not only meaningless but also misleading; fails to mention they found more papers rejecting AGW than say humans are primarily responsible

Cooking the books: John Cook's new survey is not only meaningless, but is also highly misleading because it fails to mention that they found more scientific publications whose abstracts reject global warming than say humans are primarily responsible! Only 65 out of a total of 12,271 abstracts endorsed the position that man is primarily responsible for climate change. That would indicate a 0.5% non-consensus on man-made global warming.

Reblogged from De staat van het klimaat:

Cook’s survey not only meaningless but also misleading


The new Cook et al. survey of the scientific literature is attracting worldwide media attention and even the American president tweeted (“Ninety-seven percent of scientists agree:#climate change is real, man-made and dangerous.”) about it. Yesterday my own first reaction on twitter was: “All this talking about consensus http://iopscience.iop.org/1748-9326/8/2/024024/pdf/1748-9326_8_2_024024.pdf … doesn’t improve our understanding of climate even a tiny bit” And I added that “it shows again how meaningless this kind of consensus analysis is”.
Now I still support the label “meaningless” but today I will add “misleading” to the list.
So what’s all the fuzz about? Cook et al. selected around 12,000 scientific abstracts that contained the words “global warming” or “global climate change” published in the period 1991-2011. With a large group of volunteers they then rated the papers using 7 categories. Around 8000 of the abstracts (2/3) take no clear position on Anthropogenic Global Warming (AGW). Of the remaining ~4000 abstracts more than 97% “endorse AGW” according to the paper. Only a tiny amount (78 papers) “reject AGW”. Hence they claim again that there is a consensus, that the debate is over and also that there is a gap between scientists and the public (see graph above). A much larger percentage of the scientists “endorses AGW” than the public at large.
Misleading
Now here comes the misleading part. If an abstract/paper “endorses AGW”, what would this mean for most people? Let’s look again at the tweet of Obama: “#climate change is real, man-made and dangerous”. If this is what it means for the president of the US, it probably means the same for many citizens who heard the news in the media. However, can this be sustantiated by the survey results? In no way.
To the credit of the researchers they made all their results available in a searchable database. Their rating system is online as well. There are 7 levels of endorsement, going from quantified endorsement of AGW all the way down to a quantified rejection of AGW. Seems fair enough. But here  is the issue. Only the first category can be regarded as a real or strong endorsement of AGW. Here is the description of category 1:
1. Explicit Endorsement of AGW with quantification
1.1 Mention that human activity is a dominant influence or has caused most of recent climate change (>50%).
1.2 Endorsing the IPCC without explicitly quantifying doesnt count as explicit endorsement – that would be implicit.
Now specifically look at 1.1. This comes close to the iconic statement from the IPCC AR4 report which said that “Most of the observed increase in global average temperatures since the mid-20th century is very likely due to the observed increase in anthropogenic greenhouse gas concentrations.” Now if 97% of the abstracts would repeat in slightly varying terms this major conclusion, than at least the conclusion of the survey would be more or less fair. However the survey doesn’t come even close.
Brandon Shollenberger, who is guest blogger at The Blackboard, was the first who reported that actually only 65 papers have been rated “category 1″. Yes that’s right, only 65 abstracts clearly “mention that human activity is a dominant influence or has caused most of recent climate change (>50%)”. 65 on a total of 12,000 is 0.5%. So a completely fair conclusion from their survey is that only 1 in 200 abstracts explicitly mentioned that humans are dominating climate. If you ignore the 8000 papers that were labelled category 4 (neutral, meaning having no position on AGW) the 65 would be 1.6%. The paper reported that only 78 papers (1.9% if you ignore the 8000 neutral abstracts) rejected AGW.
Now where is the 97% endorsement of AGW coming from? What the authors did is to add up the numbers of category 1 to 3 and of the category 5 to 7 which I show below*:
Category 1: 65
Category 2: 934
Category 3: 2933
Category 4: 8261
Category 5: 53
Category 6: 15
Category 7: 10
Total:         12271
As you can see the 78 “rejection of AGW” abstracts are the added number of category 5-7. Category 1-3 together adds up to 3932 papers. This 3932 divided by 4010 (the total of category 1-3 + 5-7) gives their impressive 97% (with the above numbers it is even 98%, see * for more explanation about how these numbers were found). However of these 3932 abstracts 2933 (75%) fall in category 3. Now how strong is the endorsement of AGW in this category? Here is the description:
3. Implicit Endorsement of AGW
3.1 Mitigation papers that examine GHG emission reduction or carbon sequestration, linking it to climate change
3.2 Climate modelling papers that talks about emission scenarios and subsequent warming or other climate impacts from increased CO2 in the abstract implicitly endorse that GHGs cause warming
3.3 Paleoclimate papers that link CO2 to climate change
3.4 Papers about climate policy (specifically mitigation of GHG emissions) unless they restrict their focus to non-GHG issues like CFC emissions in which case neutral

3.5 Modelling of increased CO2 effect on regional temperature – not explicitly saying global warming but implying warming from CO2
3.6 Endorsement of IPCC findings is usually an implicit endorsement. (updated this so it is more than just reference to IPCC but actual endorsement of IPCC)
I like 3.2: “endorse that GHG’s cause warming”. I also strongly agree with this part of 3.5: “implying warming from CO2″. The meaningless result of their whole exercise is that 75% of the abstracts that say something about AGW at all “link CO2 to climate change” or “imply warming from CO2″.
The misleading part is that they didn’t specify this result in their paper. Nowhere in their paper or in the supplementary material they even mentioned the total numbers in the different categories like I did in the simple table above. They only showed the total of category 1-3 in their figure 1(a):

Even the other >24% of AGW endorsement (based on the 4010) in category 2 is pretty meaningless:
2. Explicit Endorsement of AGW without quantification
2.1 Mention of anthropogenic global warming or anthropogenic climate change as a given fact.
2.2 Mention of increased CO2 leading to higher temperatures without including anthropogenic or reference to human influence/activity relegates to implicit endorsement.

In a comment under his post Shollenberger nicely explains who the different categories skew the result towards “endorsement of AGW”:
To give an example, if we say, “Humans are responsible for 40% of global warming,” that puts us in the bottom category. Change the number to 60%, and suddenly we’re in the top category. But what if we don’t give a number at all? 

If we just say “Humans cause some global warming,” we could be supporting a value 20% or 90%. Despite being able to support either position, we’d land in the top categories. That means the results will automatically be skewed toward the top.

Self confirming Now for anyone who reads climate papers frequently this is totally obvious. Climate scientists have to frame their research in the abstract and there wouldn’t be so many climate papers if there was no concern for CO2.
So the whole result of this survey is completely self confirming. Because there is a concern for CO2 there is a lot of funding of climate science. This then generates a lot of climate science papers (they surveyed 12,000 but mention there are many more). In the abstracts scientists refer to the concern about CO2. The abstract then falls into category 2 or 3 and therefore almost all the papers “endorse AGW”.

* This is how you can replicate the numbers. Go to http://www.skepticalscience.com/tcp.php?t=search and type just a single letter in the search term box (I used “a”). This generates the total of 12,271 abstracts. Then select the whole period 1991-2011 and just search the 7 different categories. This generates the table from the blog post above.

65 comments:

  1. http://wattsupwiththat.com/2013/05/21/cooks-97-consensus-study-falsely-classifies-scientists-papers-according-to-the-scientists-that-published-them/

    ReplyDelete
  2. http://rankexploits.com/musings/2013/bias-author-survey-pro-agw/

    ReplyDelete
  3. http://wattsupwiththat.com/2013/05/22/the-collapsing-consensus/

    ReplyDelete
  4. http://www.thenewamerican.com/tech/environment/item/15457-global-warming-consensus-cooking-the-books

    ReplyDelete
  5. http://notrickszone.com/2013/05/23/dana-nuccitelli-refuses-reality-richard-tol-calls-john-cooks-survey-silly-idea-poorly-implemented/

    ReplyDelete
  6. http://jennifermarohasy.com/2013/05/ten-of-the-worst-climate-research-papers-5-years-on/

    ReplyDelete
  7. http://rankexploits.com/musings/2013/climate-science-porn/

    ReplyDelete
  8. From last link above:

    Climate Science Porn

    23 May, 2013 (13:31) | Data Comparisons Written by: Brandon Shollenberger

    There’s no way to define it. We just know it when we see it. That’s how Supreme Court Justice Potter Stewart described porn. And that’s what John Cook and pals were looking for.

    I kid you not. After all our discussion of what the “consensus” is, or what it even means, it turns out the answer was in the leaked Skeptical Science forum all along. In a topic titled, “Defining the scientific consensus,” John Cook explains:

    Okay, so we’ve ruled out a definition of AGW being “any amount of human influence” or “more than 50% human influence”. We’re basically going with Ari’s porno approach (I probably should stop calling it that which is AGW = “humans are causing global warming”. Eg – no specific quantification which is the only way we can do it considering the breadth of papers we’re surveying.

    That’s right. They were completely aware of the two possible consensus positions. They made dozens of posts talking about them. Dana Nuccitelli even wanted to discuss both.

    The way I see the final paper is that we’ll conclude ‘There’s an x% consensus supporting the AGW theory, and y% explicitly put the human contribution at >50%

    But instead, they rejected any sort of scientific process in favor of hand-waving value judgments. They intentionally rejected all explicit or clear definitions for their “consensus” in favor of the “porno approach.”

    So now we know. 97% of climate science is porn.

    ReplyDelete
  9. http://climateaudit.org/2013/05/24/undercooked-statistics/

    ReplyDelete
  10. http://www.timesunion.com/business/press-releases/article/Only-65-Scientists-of-12-000-Make-up-Alleged-97-4552209.php

    ReplyDelete
  11. Roundup:

    http://www.climatedepot.com/2013/05/28/more-woes-warmist-john-cooks-97-consensus-study-falsely-classifies-3-more-scientists-dr-morner-soon-carlin-plus-round-up-of-analyses-of-cooks-study/

    ReplyDelete
  12. http://news.heartland.org/newspaper-article/2013/05/29/climate-alarmists-caught-doctoring-97-percent-consensus-claims?utm_source=feedly

    ReplyDelete
  13. Cook’s degree, which he does state properly further down in his CV is a plain B.Sc. The with honors thing is not part of the degree (usually stated as B.SC. with honors in X). The University of Queensland does not offer a B.Sc.H. degree. Cook made it up to aggrandize himself.
    http://www.uq.edu.au/study/program.html?acad_prog=2030

    Jack Maloney says:
    May 29, 2013 at 6:34 pm
    Just curious: I know there is a Dr. John S. Cook at U Queensland, but his biography doesn’t quite match up with that of the SkS blogger. Is the John Cook of Cook et al. (2013) the same person as John Cook of SkS? And is he truly a “Dr.”?

    John Cook of SkS is the same as Cook et al. (2013). And no, he’s not a doctor as my post above indicated. He’s has a three-year undergrad in basic science with a Physics major. And judging from this, looks like he’s going for his masters in psychology, due 2014.
    http://www.psychology.uwa.edu.au/research/postgrads?profile/1/id/3399

    ReplyDelete
  14. http://rankexploits.com/musings/2013/the-saga-continues/?utm_source=feedly

    ReplyDelete
  15. http://www.populartechnology.net/2013/06/the-statistical-destruction-of-97.html?utm_source=feedly

    ReplyDelete
  16. ERL: "Corrections were made to this article on 31 May 2013. A data file was added to the supplementary data."

    Monckton: "Its definition of the “consensus” it claimed to have found was imprecise: that “human activity is very likely causing most of the current anthropogenic global warming”."

    My guess is that ERL has altered what Monckton quoted to:
    "...human activity is very likely causing most of the current GW (anthropogenic global warming, or AGW)."

    Has any one a copy of the original? --AGF

    ReplyDelete
  17. http://nigguraths.wordpress.com/2013/06/08/why-the-cook-paper-is-bunk-part-i/

    ReplyDelete
  18. http://wattsupwiththat.com/2013/06/24/quantifying-the-consensus-on-global-warming-in-the-literature-a-comment/

    ReplyDelete
  19. more on Cook's lies:

    http://notalotofpeopleknowthat.wordpress.com/2013/06/30/getting-to-the-bottom-of-cooks-97-lie/

    ReplyDelete
  20. http://www.populartechnology.net/2013/05/97-study-falsely-classifies-scientists.html

    ReplyDelete
  21. http://news.heartland.org/newspaper-article/2013/06/18/climate-alarmists-caught-doctoring-97-percent-consensus-claims

    ReplyDelete
  22. https://twitter.com/RichardTol/status/357415825256361984

    ReplyDelete
  23. http://junkscience.com/2013/07/20/warmist-admits-recent-97-consensus-survey-paper-is-bogus/

    ReplyDelete
  24. http://www.populartechnology.net/2013/06/cooks-97-consensus-study-game-plan.html

    ReplyDelete
  25. http://rankexploits.com/musings/2013/tol-on-bad-papers/

    http://tomnelson.blogspot.com/2013/07/warmist-mike-hulme-97-consensus-article.html

    ReplyDelete
  26. http://judithcurry.com/2013/07/26/the-97-consensus/

    ReplyDelete
  27. http://wattsupwiththat.com/2013/07/28/on-the-97-percenters-you-must-admit-they-were-careful/

    ReplyDelete
  28. http://notalotofpeopleknowthat.wordpress.com/2013/07/29/who-are-the-97/

    ReplyDelete
  29. http://wattsupwiththat.com/2013/07/29/what-is-cooks-consensus/

    ReplyDelete
  30. http://www.climate-resistance.org/2013/07/tom-curtis-doesnt-understand-the-97-paper.html

    ReplyDelete
  31. http://judithcurry.com/2013/07/27/the-97-consensus-part-ii/

    ReplyDelete
  32. http://www.climate-resistance.org/2013/08/science-without-an-object.html

    ReplyDelete
  33. http://tallbloke.wordpress.com/2013/08/08/mike-hulme-after-climategate-never-the-same/

    ReplyDelete
  34. http://wattsupwiththat.com/2013/08/20/quote-of-the-week-about-that-97-consensus/

    ReplyDelete
  35. http://wattsupwiththat.com/2013/08/28/cooks-97-climate-consensus-paper-crumbles-upon-examination/

    ReplyDelete
  36. http://notrickszone.com/2013/08/28/lomborg-on-cook-97-survey-it-turns-out-they-have-done-pretty-much-everything-wrong/

    ReplyDelete
  37. http://joannenova.com.au/2013/08/richard-tol-half-cooks-data-still-hidden-rest-shows-result-is-incorrect-invalid-unrepresentative/

    ReplyDelete
  38. http://wmbriggs.com/blog/?p=8935

    ReplyDelete
  39. http://wattsupwiththat.com/2013/09/03/cooks-97-consensus-disproven-by-a-new-paper-showing-major-math-errors/

    ReplyDelete
  40. http://www.thegwpf.org/content/uploads/2013/09/Montford-Consensus.pdf

    ReplyDelete
  41. http://news.heartland.org/newspaper-article/2013/08/28/ipcc-lead-author-debunks-asserted-97-percent-consensus

    ReplyDelete
  42. http://www.bishop-hill.net/blog/2013/9/3/benestad-et-al-rejected.html

    ReplyDelete
  43. http://link.springer.com/article/10.1007/s11191-013-9647-9

    ReplyDelete
  44. http://wattsupwiththat.com/2013/09/03/consensus-what-consensus-2/

    ReplyDelete
  45. http://www.globalwarming.org/2013/09/05/consensus-shmensus/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+globalwarmingorg+%28GlobalWarming.org%29

    ReplyDelete
  46. http://www.bishop-hill.net/blog/2013/9/9/the-no-response-response.html

    ReplyDelete
  47. http://wattsupwiththat.com/2013/09/09/97-climate-consensus-denial-the-debunkers-debunked/

    ReplyDelete
  48. http://www.freerepublic.com/focus/f-news/3065058/posts

    ReplyDelete
  49. http://junkscience.com/2013/09/13/warmist-consensus-found-to-be-just-a-pr-campaign-climategater-admits-cook-paper-a-damp-squib/

    ReplyDelete
  50. http://www.bishop-hill.net/blog/2013/9/13/in-the-australian.html

    ReplyDelete

  51. The Viscount Monckton has written to the editor of Environmental Research Letters and asked that the 97% consensus paper by Cook et al. be withdrawn due to its sheer mendacity. I have received a copy of Monckton's very thorough letter and offer some excerpts from it below:

    The paper claimed a 97.1% “scientific consensus” among the abstracts of 11,944 climate change papers published from 1991-2012. The true “consensus” was not 97.1%. It was 0.3%.

    The defective paper’s introduction said: “We examined a large sample of the scientific literature on global CC [the abstracts of 11,944 papers on climate change], published over a 21 year period [1991-2012], in order to determine the level of scientific consensus that human activity is very likely causing most of the current GW (anthropogenic global warming, or AGW)”.

    Computerized and manual examination by Legates et al. (2013) of the authors’ data-file, made available only some weeks after the paper had appeared, showed that on that file the authors had marked as few as 64 abstracts out of 11,944 (0.5% of the entire sample) as explicitly endorsing that “scientific consensus” as defined in the introduction to their paper. Legates et al., on further examining the 64 abstracts, found that only 41 of them, or 0.3% of the entire sample, had in fact explicitly endorsed that “scientific consensus”.

    However, the defective paper you published concluded with these words: “Among papers expressing a position on AGW, an overwhelming percentage (97.2% based on self-ratings, 97.1% based on abstract ratings) endorses the scientific consensus on AGW.”

    The authors had stated at the outset their intention to determine the level of “scientific consensus that human activity is very likely causing most of the current GW (anthropogenic global warming, or AGW)”. They had listed this standard, quantified definition of “scientific consensus” in their paper as

    “(1) Explicit endorsement with quantification (explicitly states that humans are the primary cause of recent global warming)”, the first of seven “levels of endorsement” to which they assigned the abstracts.

    Yet they did not disclose in their paper how few abstracts – just 64 – they had marked as having stated support for that standard, quantified “scientific consensus”.

    To conceal how very small this number was, they added together all of the abstracts they had assigned to the first three of their seven categories, treating all three categories as one, and did not state the three values separately. An impartial peer reviewer would have spotted this.

    The seven categories or “levels of endorsement” listed in the paper, with the abstracts marked on the data file or disclosed in the paper as falling within each category, were – Level of endorsement of “scientific consensus” in 11,944 abstracts

    Marked

    1 Explicitly states that humans are the primary cause of global warming 64
    2 Explicit endorsement without quantification [We cause some warming] 922
    3 Implicit endorsement 2910
    4a No opinion 7930
    4b Uncertain 40
    5 Implicit rejection 54
    6 Explicit rejection without quantification 15
    7 Explicit rejection with quantification 9
    Total 11,944

    So Cook et al. totally misrepresented their own data

    ReplyDelete
  52. http://judithcurry.com/2013/09/17/consensus-denialism/

    ReplyDelete
  53. http://wattsupwiththat.com/2013/09/20/join-my-crowd-sourced-complaint-about-97-1-consensus/

    ReplyDelete
  54. http://junkscience.com/2013/09/26/ipcc-reviewer-but-to-be-honest-theres-not-a-clear-consensus-among-the-scientific-community/

    ReplyDelete
  55. http://dailycaller.com/2013/09/04/report-there-is-no-97-percent-global-warming-consensus

    ReplyDelete
  56. http://clima-virtual-vs-real.blogspot.com/2012/06/scientific-method-against-anthropogenic.html

    ReplyDelete
  57. http://joannenova.com.au/2013/10/catalyst-says-consensus-wrong-on-cholesterol-but-unquestionable-on-climate/

    ReplyDelete
  58. http://judithcurry.com/2013/11/10/the-52-consensus/

    ReplyDelete
  59. http://wattsupwiththat.com/2013/11/20/the-97-consensus-myth-busted-by-a-real-survey/

    ReplyDelete
  60. http://blog.heartland.org/2013/11/the-latest-meteorologist-survey-destroys-the-global-warming-climate-consensus/?utm_source=feedly

    ReplyDelete
  61. http://theclimatescepticsparty.blogspot.com/2013/12/cook-responds-badly.html

    ReplyDelete
  62. http://nigguraths.wordpress.com/2013/12/09/tom-curtis-an-open-letter/

    ReplyDelete
  63. http://hiizuru.wordpress.com/2013/11/23/consensus-is-god/

    ReplyDelete
  64. Protip: Science is not consensus.

    ReplyDelete
  65. http://wattsupwiththat.com/2014/09/08/new-paper-fraud-bias-public-relations-the-97-consensus-and-its-critics/

    ReplyDelete