Saturday Sermon: How To Avoid P-Hacking

Outsourced to Daniel MacArthur:


A couple of years ago, it was discovered that two very prominent economists, who based on erroneous data (which was also not very robust), were telling governments that when debt reaches ninety percent of GDP (or more), economic growth decreases dramatically. This was used by the austerians to argue that deficit spending wasn’t needed–an incredibly stupid policy.

At the time, I felt they did a piss-poor job of validating their data:

From what I’ve been able to gather, many economists think this is the least problematic error, that the decision to exclude certain data is more problematic, but I think that’s completely wrong. As best as I can tell, the spreadsheet is about twenty columns (max) by fifty rows. Somewhere around 1,000 cells. There’s nothing wrong with small datasets, but how the fuck do you not check every goddamn cell when the dataset is that small? In genomics, the summary table for some of the things we do has half a million cells. It’s challenging to check for errors, but that’s the job. Hell, yesterday I gave a talk where I alluded to a one-in-a-million error rate–and some physicists and engineers probably think we’re rank amateurs with that error rate (if you sequence the same exact strain of E. coli twice, that error rate [one in a million] yields ten false differences between the two strains; science is hard). In science, rigor is everything–you can’t be any smarter than your data are accurate. No field can allow itself to be this sloppy, especially when the stakes are that high.

And I still don’t.

This entry was posted in Economics, Genomics, Statistics. Bookmark the permalink.

4 Responses to Saturday Sermon: How To Avoid P-Hacking

  1. Min says:

    Not only that. The 90% figure came from their division of their data into less than 30%, 30 – 60%, 60 – 90%, greater than 90%. If you look at a plot of the data, there is nothing special about 90%. They made it sound like the edge of the world.

  2. jrkrideau says:

    My feeling was they got the results they wanted. It was not research as such, just a confirmation study of a political viewpoint.

    Of course, anyone stupid enough to use a spreadsheet like Excel for serious analysis … Words fail.

  3. rwv says:

    There’s actually a lot one can do in a spreadsheet, allthough I’d prefer to use open software..
    When it comes to austeritarians, in a personal conversation I once had with one of them, I was told anything and everything can be proven scientifically if one wants to. (ironically this was a response to arguments I made against auterity measures)

    Fact is -that without the study you refer to- these people would have cherrypicked their results from elsewhere just as easy.

  4. rwv says:

    just as easily; my bad!

Comments are closed.