What about the replication crisis? It's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I can't tell you how many times I've seen headlines or abstracts of studies that seem to contradict ones I've seen previously, and back and forth! Particularly in the social sciences.
I recall one study that said all white people are committing environmental racism against all non-white people. I dove in and read the whole thing wondering what method could have yielded scientific confidence in such a broad result. Turns out the model used was a semi-black box that required a request for access and a supercomputer to run. But it was in a Peer Reviewed Scientific Journal and had lots of Graduate Level Statistics so I guess it seemed trustworthy.
The issue here has not much to do with the replication crisis. It has to do with the fact that most people who use bits of information to make their point more convincing don't care whether that information is true or not. They are not seeking to convince the other side of the issue, they are seeking to convince other believers.
It is literally like this:
- someone makes a point that questions your believe
- you google a phrase that would come in studies that proof otherwise
- you take the first thing that looks promising, and fly over the first page, and paraphrase a good bit in a way that makes your point
- you publish it as part of a post, youtube video or whatever
- danger averted
Bad studies play into this, but even if the studies are good, or bad studies that have been retracted the same thing happens. James Wakefield who originally published the "combined vaccines cause autism study" after patenting a non-combined measles vaccine had his study retracted by the lancet soon after publication. He lost his status as a doctor etc. And you will still find people who use his study as a source.
Of course studies whose outcome collide with our believe systems are always harder to trust than those who validate it — but this is why you look at the methods used and other indicators that might make that study bogus.
A replication crisis indeed exists. All the more reason to analyze rigorously. Poor analyses (and borderline name-calling) in the original article do not help with the crisis.
>it's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods).
I don't think this is true. It is possible to put a lot of work into unsound statistics and to make a lot of "noise and fury" about how mathematical you are while failing some basic principle, but I don't think sound statistics can mislead. The replication crisis was caused by scientists not being rigorous and journals not forcing them to be. You absolutely cannot accept publication as a sign of sound techniques except in journal/field combinations that have a deserved reputation.
Of course they can, unless you magically exclude all statistics that made a bad assumption on independence.
I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics.
Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on forecasted cold days.
Is that the fault of statistics though, or the non-statistical implication of causation that was tacked on the end of the statistical detection of correlation? Statistics is pretty explicit that it can't tell you about causality, right?
> It's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods)
The book "How to lie with statistics" is one of the best statistics textbooks that I have read. It basically makes you immune to misleading stats (charts, tables, everything).
IIRC, the only thing that is missing from the book (it's a really old book) which is very relevant is p-hacking.
Given the explosion in the number of journals and the impossibility of effective peer review, being published in a journal does not mean what it used to. This is part of the material drivers for the replication crisis (journals can no longer effectively gatekeep scientific validity), but it also reflects something real about the practice of science: little social cliques come up with pet theories and, over time, "fight" with these theories on epistemic common ground. The successful ones, we'd like to think, are the ones that last the most rounds in the fight, but that probably only holds in the long run. Contradiction, in itself, is normal (and was before!)
I recall one study that said all white people are committing environmental racism against all non-white people. I dove in and read the whole thing wondering what method could have yielded scientific confidence in such a broad result. Turns out the model used was a semi-black box that required a request for access and a supercomputer to run. But it was in a Peer Reviewed Scientific Journal and had lots of Graduate Level Statistics so I guess it seemed trustworthy.