Statistics are the magic beans of pop science. Tell someone that heavy drinkers live longer than teetotallers, and she'll say, "How is that possible? What about cirrhosis of the liver?" Tell her there are statistics proving it, and she'll say, "Wow, I never would have guessed that!" That particular stat is making the rounds of the news talk shows right now, and I'd imagine so are a lot of beers.

The first problem with relying on statistics is that most people don’t understand the math very well. To someone trying to prove a point, stats are like a buffet of numbers. For true believers, confirmation bias takes over. Even dispassionate scientists don't always interpret the numbers correctly.

Odds Are, It’s Wrong. Science fails to face the shortcomings of statistics

It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.

Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.

Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.

“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”

Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”

Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense.”

Between poor understanding of the math, confirmation bias and the need to publish and perish, it is no wonder that the science establishment insists on peer review and replication of results. But even then, could something go wrong? Yes, the data being sampled may be narrow or skewed.

The Weirdest People in the World?
To be published in Behavioral and Brain Sciences

It’s about a truly unusual group: people from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. In particular, it’s about the Western, and more specifically American, undergraduates who form the bulk of the database in the experimental branches of psychology, cognitive science, and economics, as well as allied fields (hereafter collectively labeled the “behavioral sciences”). Given that scientific knowledge about human psychology is largely based on findings from this subpopulation, we ask just how representative are these typical subjects in light of the available comparative database. How justified are researchers in assuming a species-level generality for their findings? Here, we review the evidence regarding how WEIRD people compare to other populations.

The Behavioral Sciences Database is Narrow ... a randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the West.

Researchers Often Assume their Findings are Universal ...despite their narrow samples, behavioral scientists often are interested in drawing inferences about the human mind and human behavior. ... Leading scientific journals and university textbooks routinely publish research findings claiming to generalize to “humans” or “people” based on research done entirely with WEIRD undergraduates.

[Update & thanks to DF]
Papers by Dr Andrew Wakefield led many doctors and parents to see a correlation between MMR vaccines and increasing frequency of autism in children. Research by Dr Mark Geier, and his son, seemed to implicate the vaccine preservative Thiomersal, which contains mercury, as well. Wakefield's study spurred the withdrawal of MMR vaccines, and vaccines containing Thiomersal were also withdrawn as a precaution.

But eventually, groups like the American Academy of Pediatrics, criticized the Geiers' research as having, "numerous conceptual and scientific flaws, omissions of fact, inaccuracies, and misstatements." Others cited "methodological problems in his research including not presenting methods and statistical analyses to others for verification, improperly analyzing data taken from Vaccine Adverse Event Reporting System, as well as either mislabelling or confusing fundamental statistical terms in his papers, leading to results that were "uninterpretable"."

Wakefield went beyond poor methodology and secrecy. As reported by Brian Deer, he was paid to attack MMR vaccines to support class action litigation that was planned by a UK solicitor. Even his subjects were paid by the solicitor. Geier has been somewhat discredited; Wakefield is no longer a physician.

As described by Tim Lambert on ScienceBlog, a scary combination of narrow samples, bad methodology and lack of ethics has been seen in John Lott's work to promote his theories that gun ownership leads to lower crime rates.

John Lott has claimed that he conducted a survey over three months in 1997 that found that in 98% of defensive gun uses the defender merely has to brandish the gun to break off an attack. It is almost certain that he never did a survey because:

* nine published surveys give numbers ranging from 21% to 67% as to how often defenders shoot, far more than the 2% Lott claims
* Lott claims that his survey found defenders firing in 2 out of 28 cases, which is 7%, not 2%. Nor is it possible that the weighting scheme Lott now claims to have used turned 7% into 2%.
* no evidence that Lott ever conducted such a survey can be found
* Lott has repeatedly changed his story about the source of the 98% figure, variously attributing it to "national surveys" and some particular polls, only publishing the story about the survey in 2000.
* Lott made the 98% claim on Feb 6, 1997, well before his survey was completed.
* Lott changed his story about the survey.
* Despite extensive coverage on the net, in many papers such as the New York Times, the Chicago Tribune and the Washington Post, an ad in the alumni magazine and in the best-selling book Freakonomics, none of the eight students Lott claims conducted the survey have been heard from.
* Lott claims to have "replicated" his survey with a new one that gives 95% brandishing, when in fact the new survey found only seven gun users (one of whom fired), far too small a sample to yield a meaningful brandishing number. (And if you correct his arithmetic you get 91% anyway.)

In 1998, John Lott published a book entitled More Guns, Less Crime. In that book he presented statistical evidence that concealed-carry laws were associated with lower crime rates.

In 2002, Ian Ayres and John Donohue analysed a more extensive data set and found that, if anything, concealed carry laws lead to more crime. Lott responded with a new analysis that he claimed confirmed the "more guns, less crime" hypothesis. Ayres and Donohue's response (April 2003) was devastating—Lott's data contained numerous coding errors that, when corrected, eliminated the results and, this was the second time these sort of errors had been found in Lott's data.

More at Reason.com:

The first round of dispersed investigation came when a Minnesota attorney named David Gross came forward to say he had been the subject of a survey that sounded like Lott's. The Washington Times ran a brief story implying that the question about Lott's survey was now closed.

 

But bloggers were more skeptical: Gross turned out to be a gun rights activist himself, with the group Concealed Carry Reform, NOW! Historian Thomas Spencer unearthed a letter to the Minneapolis Star Tribune in which Gross wrote that gun control advocates "dance on the graves of the innocent victims and glory in their spilled blood." Another blogger, the pseudonymous Atrios, found news reports recounting how Gross had taken over the names of several gun control groups that had neglected to renew their corporate status with the state.

Of course, a gun activist would be the most likely to hear about the controversy and come forward. But skeptics continued to doubt Gross' account, and some of Lott's former defenders wrote that they wished for some further independent confirmation of his account. Lott, currently a resident scholar at the American Enterprise Institute, has conducted a new survey, which he includes in his new book, The Bias Against Guns (Regnery).

Meanwhile, several of the bloggers who had been writing about the controversy -- a group that included me -- drew the ire of someone called Mary Rosh. Rosh, who identified herself as a former student of Lott's who had long admired his fairness and rigor, said that it was irresponsible to post links to the survey debate without calling Lott first. This sounded odd, not only because bloggers very seldom do that kind of background research before posting a link, but because Lott had made precisely the same criticism several times in e-mails to bloggers covering the story.

A Google search revealed that Rosh had for several years been a prolific contributor to Usenet forums, where she regularly and vociferously defended the work of Lott. On a whim, I compared the I.P. address on Rosh's comment to the one on an e-mail Lott had sent me from his home. They were the same.

Lott admitted to using the sockpuppet on usenet, websites and Amazon reviews.

Despite all this, John Lott continues to be cited - at gunfacts.info.