Documents

5 Statistical Sins

Description
Belongs to their owner
Categories
Published
of 18
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  5 STATISTICAL SINS (AND HOW TO AVOID THEM)  Most published scientific results are false.That's the thesis argued–convincingly–by John Ionnadis in his landmark 2005 paper.Here's one reason for this.Imagine that you're living in an alternate past. The year is 1850. It’s the height of the California gold rush. Except, in this timeline, you’re living in a more scientifically literate society, so researchers set out to prove that there are ways to more e ! ciently strike it rich. An agenda I can get behind.These researchers try whatever they can think of–testing, to name a few, the divine power of prayer, sacrificing groundhogs, and dowsing. None of these pan out (heh).Then, one otherwise uneventful Tuesday, a paper is published, Human echolocation of gold veins: evidence of an adaptive mechanism for aural min-eral detection. The paper describes an experiment where researchers split 1000 gold-hungry migrants into two groups: a traditional control and an echo-location group. The echolocation group was instructed to walk around clack-ing their tongues in pursuit of the shiny yellow stu   .Here's the crazy part: the echolocation people found a statistically signifi-cant amount of excess gold. The authors speculate that humans have a hereto-fore unknown sensory organ that picks up mineral densities. This, they say, was useful in the ancestral environment for tasks like finding salt, copper, and primo soon-to-be farmland.Everyone reads the paper. It's flawless, compelling. I’d call it a slam dunk but basketball won’t be invented for another 40 years. Soon, only fools search for gold without clacking their tongues madly, harnessing this power of hu-man echolocation. I. Except, you know, homo sapiens (unfortunately) have no particular knack for echomining. So what went wrong? Why is this paper false, even though the researchers executed the scientific process the right way? 5 STATISTICAL SINS1  Here's one possible answer: those researchers weren't the only ones that tested echolocation. There was a lot of excitement–gold to be had! Anything and everything was tried.For the researchers who found nothing other than the null result, no pa-per was published. Of course humans can't detect gold with tongue clacking. This is not publication-worthy science. But, in a universe where enough people test something, someone is bound to obtain a significant result. The laws of probability dictate it. If 20 teams tested human echolocation, the odds of one team reporting a false positive are not 5%, like you might expect with a p-value threshold of .05, but 65%. II. This concept is called the file drawer problem, because studies that find expected, business-as-usual results are relegated to the metaphorical file drawer, away from the light of the fair sun.This is a real problem, with real consequences. The antidepressant Re-boxetin, for instance, was approved for use after clinical trials found it e   ec-tive. One problem. The stu    doesn’t work. It was publication bias on the part of manufacturer Pfizer. Pfizer ran many studies, publishing only the positive findings.Something similar, but less nefarious, happens with the study of extra-sensory perception–you know, psychics, mind control, that kind of thing. When a study finds that people can't read minds (duh), the paper doesn't get either written up or published.But when such a study finds a positive result, everyone wants to read it. There is a whole field like this, parapsychology, and many of the papers it en-compasses are at least as fanciful as my echolocation example.Like JB Hasted's paper, Paranormal Metal Bending , where he reports that participants could bend metal with their minds. It contains a photo of a glass globe filled with paranormally-scrunched paper clips. Notably, there is hole in the globe, but Halsted explains, We have found it necessary that a small orifice be left in the glass globes in which wires are bent.”Right. 2  Amusingly, academic finance has the opposite problem. If you do dis-cover an anomaly in, say, the stock market, you don't publish it. You arbitrage the opportunity away, making millions in the process. III. This is one of the reasons why most published research is false. In the coming pages, I'll detail 5 more common mistakes, and what you can do to avoid them. Mistakes I’ve Made These are all errors that I can recall making myself at one point or an-other. Mistakes I'm sometimes embarrassed to admit ever happened, but they did and are, in the case of my GitHub repositiories, a matter of public record.Yeah, I've been fooled a lot of times–to name a few: The depletion model of willpower.  I read this book by Roy Baumeister, and then later saw him speak. It all made sense to me–when you do some-thing hard, your brain uses up some willpower, and that needs replenished, maybe via a sugary drink.But this has failed to replicate a couple of times. The most damning of which is a study by Veronika Job, Carol Dweck, and Greg Walton, which found that the depletion model of willpower only a   ects those who believe in it. It's all in your head. Whoops. Happiness tipping points.  Okay, this one is embarassing, because I really should have known better. Barbara Frederickson and Marcial Losada came out with this pretty neat paper in 2005, called “Positive A   ect and the Complex Dynamics of Human Flourishing”.Basically, it argues, that above a certain ratio of positive to negative emo-tions, there exists a tipping point where humans flourish. So, if you have 2 good moments for every bad moment, you're kinda miserable. But if you have 3 good moments, you're thriving–like the Dalai Lama or something.So, the red-flag that I should have spotted in this paper is that the authors peg that tipping point at 2.9013. Yeah. You're never going to get a value that precise in psychology. 3
Search
Tags
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks