Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How is this different from a clinician who has to diagnose autism without even such a handy instrument.


Colloquial view: Presumably someone going to see a psychiatrist about autism in interested in an accurate outcome as well as the doctor.


The main test they use for ASD, the Autism Diagnostic Observation Schedule (ADOS) – a Cochrane review [1] found that across 12 studies of diagnosis in preschoolers, the ADOS-1 [2] had a summary specificity of 0.8 – in other words, an average false positive rate of 20%. But, looking at the studies individually, the specificity ranged from 0.2 to 1.0 – in other words, the false positive rate ranged from 0% to 80% across the individual studies. And, that is in research contexts – there are good reasons to believe that the test will be less accurate in clinical use than in research contexts. In order to use the ADOS in research contexts, one has to demonstrate "research reliability", which is a process of scoring videos of the test being performed and comparing your scores with one's colleagues, repeating the process until you get sufficient agreement, and doing this again once every few months to make sure you stay in agreement. None of that rigour is required in clinical use. Added to that, I think in clinical contexts psychologists sometimes feel the urge to "put their thumb on the scales", they'll think "I know the test says this child doesn't have ASD, but I know they are struggling and an ASD diagnosis will get them funding that will help them, so I'll just fudge the numbers a bit to make it come out positive." Some psychologists who would never do that in research (it's scientific fraud) will do that sort of thing clinically. So, it seems very likely (if difficult to prove rigorously) that tests like the ADOS will have less accuracy in clinical contexts than research contexts, so however accurate they are measured to be in these studies, it is probably worse than that in real life.

Sometimes, they suggest combining the ADOS with the ADI-R (Autism Diagnostic Interview-Revised), which is a structured interview of parents/caregivers. The combination is even called the "gold standard", with the suggestion that using the two in combination is more accurate than either alone. But, does the research actually support that belief? Well, that Cochrane review says they could only find one study (in preschoolers, they excluded studies of older individuals) which compared the accuracy of ADOS-1+ADI-R to ADOS-1 alone. It found a modest improvement in specificity (equivalent to false positive rate), at the cost of a modest worsening in sensitivity (equivalent to false negative rate), but the result wasn't statistically significant. So, this belief that using the combination improves accuracy doesn't appear to really be supported by the evidence.

But, do you think clinicians tell parents any of this? In my personal experience, no. Either they are ignorant of the research, or they are keeping that knowledge close to their chests. How much faith would parents put in the results of the ADOS if they were told that in research contexts it had a 20% false positive rate on average? And that there is good reason to believe that in clinical contexts (their current context) the test probably performs worse than that? And that 20% is just an average over multiple studies, and in some individual studies its false positive rate was substantially higher?

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6513463/

[2] the ADOS-1 has been replaced by a new version, the ADOS-2. At the time of that Cochrane review, there hadn't yet been any studies published on using the ADOS-2 in preschoolers, so it was excluded from their analysis. But, the fundamentals of the test haven't changed, so I doubt the ADOS-2 performs significantly better than the ADOS-1 did.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: