“Just keep doing tests. Eventually you are bound to find something.” She was right, of course. But not in the way she meant.
Frustrated by the absence of either an explanation or effective treatment for her years of chronic fatigue and nonspecific malaise, she came in requesting a long list of tests, many for things not shown to cause illness, and none of which were known to cause symptoms like hers. We had seen her multiple times for these issues over the previous four years and had - more than once - considered and tested for the most common and most plausible causes. We had also considered and tested for a long list of unlikely but possible causes. I had frankly discussed the fact that not all problems have explanations or effective treatments, and that we would continue to follow her symptoms and re-evaluate if there were changes or new medical information, but that for now we needed to focus on minimizing the impact on her quality of life by attempting to treat the worse symptoms.
She was unwilling to consider the possibility that her unhappy marriage, chronically ill father living with her, and her son’s recurrent problems with drugs and the law played a role. Instead, she wanted testing for improbable and impossible causes. (Her list included testing for malaria, African sleeping sickness, pesticide toxicity, systemic yeast.) I declined, explaining that indiscriminate testing for things that are incompatible with her symptoms would not be helpful and might be dangerous. She was (angrily) dissatisfied with my approach and found another physician to pursue the testing she wanted. I learned later from her brother that she had developed irreversible kidney failure and was on dialysis while awaiting a transplant. Did I miss kidney disease, I asked? No, he said. Her kidneys failed as a result of the treatment she received for an illness it turned out she didn’t have.
The law of large numbers and the law of truly large numbers tell us that the improbable is inevitable. This has always been true. A favorite party game is looking for people with the same birthday: with a group of 23 people, there is a 50% change of two with the same birthday. The probability rises to 99.9% in a group of 70. With a large enough sample (given enough opportunities) every possible outcome is not just a possible outcome but an inevitable outcome. This is described here and here and here (andhere for the mathematically sophisticated).
If one does enough tests, one will get an abnormal result. Because of the way ‘normal’ is defined in most medical testing, a (false) positive is nearly guaranteed if one does 20 tests.
In the era of big data, this also means there is a substantial risk of finding (and the media hyping) coincidental associations that are statistically inevitable noise, not real world causations. Consider the Baltimore stockbroker in Ellenberg’s book or the famous xkcd cartoon showing that green jelly beans cause acne, or Drosnin’s absurd claim that equidistant letter sequences (ELS) reveal hidden messages in the Bible.
The principle is: as you keep trying, the unlikely becomes first probable and then inevitable, and coincidences will outnumber causation.
As the extreme example of my patient who ended up needing a kidney transplant following treatment of statistical noise illustrates, this can be a serious problem in medicine. Less dramatic but far more common manifestations in medicine include
- The multichannel chemistry panel that runs 20 or 30 tests at once (making an abnormal that requires further testing nearly inevitable)
- Lifeline Screening of low risk and asymptomatic patients. Their web site touts as ‘evidence’ testimonials from people with chance outcomes.
- The ‘shotgun’ approach (running as many tests as possible) when confronted with a challenging medical problem. An outstanding internist and teacher at the University of Rochester where I trained was known for tracking down medical students who had ordered tests and quizzing us about how we would change treatment based on sets of hypothetical results. Woe unto the student who did not have good answers.
- The Incidentaloma.
The failure to account for ‘base rate’ effect compounds this (also here). Twenty percent of well young children in day care have strep in their throat as an incidental finding during the winter, making false positives a real problem when testing children with viral URI symptoms (runny nose and cough). Ninety percent of actively pitching asymptomatic major league starters have what appear to be surgically correctible lesions on shoulder MRIs, but no shoulder problems. (Discussed further here.)
The solution, of course, is to think more and test less.