Here's a little puzzle to see whether we really understand the significance of accuracy.
A patient goes to a doctor and after explaining the symptoms, the doctor thinks it might be illness X, for which, fortunately, there is a test. He explains that the test is 90% accurate, that is 90% of the test results are right but 10% of them are incorrect.
The patient agrees to undergo the test and a couple of weeks later the patient gets the bad news that the test was positive saying he has illness X. Now the question - should he believe the results? Is he really likely to have illness X?
Intuitively we all say of course he is likely to have the illness, because the tests are 90% accurate.
But, we are missing a vital piece of information. How many people in the population actually suffer from illness X? Since I'm making up the problem I happen to have that statistic to hand and it turns out that at any one time, there is an incidence of 2 in 1000 of the population suffering from illness X.
Now let's do some simple sums.
Let's assume there are 10,000 tests done. We know that given our statistics, twenty of them will have illness X.
Since the test is 90% accurate, 18 of them will be told correctly that they have illness X, and 2 of them will be told incorrectly that they don't.
Now let's think about the rest, the other 9980 people who in fact don't have the illness at all.
For 10% of them, the test will return incorrect results so 998 of them will be told incorrectly that they have illness X. The remaining 8978 will be correctly told that they don't have it.
Lets add up the results:
Told they have illness X: 1016
Told they don't have illness X: 8984
Number who actually have the illness: 20
Probability that someone told they have it really does: just under 2%.
Probability that someone told they don't have the illness really doesn't: a touch over 90%.
This might seem like a catch question but it's an illustration of why statistical information has to be examined very carefully. It's remarkably easy to jump to conclusions without enough information.
If it's any consolation, this test is regularly applied to journalists, scientists, academics, etc, and most of them get it wrong too.
Incidentally, that's why there has to be so much rigour and exactitude in conducting medical tests for illnesses, especially rare ones. Reliability has to be very very high to justify diagnostic conclusions. A test accuracy of 90% with an uncommon illness is almost useless! So although it sounds impressive, we are really comparing it with how we'd feel in a test we'd completed ourselves - if we scored 90% we'd feel quite pleased. Science and especially medicine has to have very much higher standards.
