James Le Fanu

Back to Research

BMJ – December 2002

The case of the missing data

1. The morning of 26 February 2000.
2. That evening.
3. A week later.
4. Footnotes.

---

“Is there any point to which you would wish to draw my attention?”

“To the curious incident of the dog in the night time.”

“The dog did nothing in the night time.”

“That was the curious incident,” remarked Sherlock Holmes.

Sir Arthur Conan Doyle. Silver Blaze ¹

---

“The term ‘non-barking dog’ refers to a species of anomaly detail that could reasonably have been expected to appear in evidential text but which, for whatever reason, is absent.”

Eric Shepherd. Non barking dogs and other odd species. Med Sci Law 1999 ²

---

(It is the morning of 26 February 2000. The famous detective Sherlock Holmes has just been joined in the breakfast room of his flat at 221b Baker Street by the ever reliable, if unimaginative, Dr Watson.)

Holmes: Why, Watson, you don’t seem your usual cheerful self this morning. Something preying on your mind? Out with it, man. I don’t want my day ruined by one of your black moods.

Watson: I am, as you so astutely observe, much vexed. You will know I have been much persuaded by those distinguished members of my profession who claim I would avoid the misfortunes of suddenly dropping dead from a heart attack were I to adopt what they like to call a "healthy lifestyle." ³ So now I start the morning with a brisk walk round Regent’s Park, have controlled my tobacco addiction with those marvellous nicotine patches, and have given up the pleasures of bacon and eggs for breakfast and of Mrs Beeton’s powerful puddings.

It must be said the value of such measures has not been confirmed by clinical trials ⁴ indeed, there is even a rumour that heart disease may be a biological phenomenon caused by a newly identified strain of bacterium ⁵ but I have always been impressed by how our American cousins have been rewarded for their self denial by a precipitous decline in the number of coronary deaths. ⁶ Or that at least is what I believed until I picked my copy of the Lancet this morning. It reports the results of a massive WHO-Monica study of trends in heart disease in 27 countriesa major enterprise indeed. These trends, it turns out, "fit poorly" with the lifestyle risk factors. ⁷ So it’s scarcely surprising I am a trifle upset.

Holmes: Calm yourself, Watson, calm yourself. This is all most intriguing. Have you, by any chance, brought that learned journal with you this morning?

Watson: Why, yes indeed but I doubt you would make head nor tail of it.

(Ten minutes elapse before Watson’s gloomy reverie is interrupted by the great detective.)

Holmes: Quite extraordinary, my dear Watson. The phenomenon this study seeks to explain the changing rates of heart disease in the recent pastcould not be more straightforward and could best be illustrated graphically so you could see at a glance what is going on. But no. Rather, we have several massiveand quite uninterpretabletables of figures reporting such things as "the average annual percentage change in coronary events over the last five years" whatever that might mean. I suspect we are being blinded by science. There may be something important and I would like to know what it might be. But first we have some elementary detective work to do. I suggest you make a brief visit to the Royal Society of Medicine library round the corner and dig out the relevant statistics.

(That evening.)

Holmes: From your expression, my dear Watson, I see your researches have been successful.

Watson: Yes indeed, but not without some difficulty. I thought there would be no problem in finding the relevant data summarised in a paper or with the help of that wonder of modern information technology, Ovid Medline. But I drew a blank. So there was nothing for it but to dig out the original figures year by year. These I have presented, as you suggested, in graphical form for men between the ages of 45 and 64. ^8-10

The picture that emerges is so dramatic it is hard to imagine how it could have been overlooked. Starting with the United States, Canada, Australia, and New Zealand (fig 1a), the rate climbs steeply throughout the 1950s to a peak of around 600 per 100,000 in the mid-1960s before falling equally precipitously year by year over the subsequent 30 years to around 200 per 100,000. This is an extraordinarily striking picture of a disease that has risen and fallen in parallel in these countries over a period of 50 years. When we turn to fig 1b we see that, after a 10 year lag, the same pattern becomes apparent in western Europe. Finally, the graph for the countries of eastern Europe (fig 1c) shows the sort of massive increase that was being recorded in the United States and Australia back in the '60s.

Holmes: Tell me, Watson, is it conceivable that this pattern which is much more striking than I could have anticipated from the Lancet paper might be accounted for by changes in the risk factors to which you have alluded?

Watson: Frankly, I doubt it. There have, it is true, been claims that changing social habits can account for the decline in heart disease, ^{11 12} but this could be an instance of the "post hoc, ergo propter hoc" fallacy. This is not the pattern of a disease strongly influenced by patterns of social behaviour were it to be so, one would have to presume that in each of these countries quite independently vast numbers of people simultaneously and consistently change their lives first in an "unhealthy" direction to account for the rise, and subsequently in a "healthy" direction to account for the fall. This would seem most unlikely and is of course precisely what the Monica study reveals has not happened.

I may be just a humble medical practitioner, but statistics as the great Sir Austin Bradford Hill once remarked requires "the application of commonsense to figures," and my commonsense instinct tells me that this dramatic rise and fall resembles the picture of a biological disease such as an infectious epidemic.

Holmes: You know, Watson, this is beginning to remind me of a case from several years ago that you will recall when the racing horse Silver Blaze disappeared from its stable in mysterious circumstances.

Watson: Well, you've lost me there. I cannot imagine what a disappearing racehorse has to do with it.

Holmes: I agree the link is not obvious, but wait and see. In the meantime I am sure there are still more surprises in store for us. If this pattern of heart disease renders the "lifestyle" theory how shall I put it? insecure, then so must be the circumstantial evidence on which it was based the cross cultural comparisons of heart disease rates between the West and Japan and the increased rates of heart disease in Japanese migrants to the United States. ^{13 14} You look surprised that I should know about these things, but in your absence I have not been idle. I have just spent a couple of hours perusing my medical reference book here in front of the fire and now have a hunch how to pursue the matter further. We can, I think, take it for granted that exercise is good for humans and smoking bad, so the contentious issue here involves the circumstantial evidence incriminating the "high fat, Western diet." We need first to test the validity of these cross cultural comparisons: do they, for example, hold for a disease whose cause is known such as smoking and lung cancer? We also need to test the Japanese migrant evidence by examining whether the pattern of heart disease in migrants who share a similar dietary pattern to the United States remains unchanged.

Watson: I will pursue the lines of inquiry you have suggested and report back soon enough.

(A week later.)

Watson: I know you have been anxious to hear the outcome of my further researches, and I can only apologise it has taken me so long.

Holmes: And what, Watson, detained you?

Watson: Well, it’s all a bit rum. The questions you posed when we last met were straightforward enough, and I expected to have little difficulty laying my hands on the relevant facts and figures. But, and this accounts for my delay, quite extraordinarily the obvious sources failed to turn up a single reference to these matters, and it was only with considerable difficulty that I managed to find anything at all.

Holmes: You mean to say there is hardly anything in the medical literature, as you like to call it, about the cross cultural association between smoking and lung cancer, or the patterns of heart disease of migrants other than the Japanese to the United States?

Watson: Precisely so.

Holmes (sotto voce): Silver Blaze rides again.

Watson: I do wish you would stop going on about that confounded horse. I can’t see what connection it can have to my arduous researches of the past week.

Holmes: You will, Watson, you will. In the meantime I presume you were eventually successful in your quest.

Watson: I was. I have here a small set of graphs as you say, much the most lucid way of presenting a lot of data and if you will draw up your chair I will take you quickly through them. We start with the cross cultural studies comparing the West and the Far East (fig 2): there is a clear dose-response relation the more meat and dairy products consumed, the higher the incidence of heart disease. ¹⁵ This seems pretty convincing, until you shade in the countries of western Europe I suppose you might call this a "within cultural" comparison and that dose-response relation disappears. Thus, the heart disease rate in Finland is four times greater than in Switzerland even though the amount of fat consumed in the two countries is virtually the same. My instinct would be to put more trust in this "negative" within cultural comparison than in the "positive" cross cultural comparison, where the number of confounding variables, as we like to call them, is likely to be so much greater.

This instinct is confirmed, as you suggested it might be, by the comparable data for smoking and lung cancer. There is, astonishingly, only one cross cultural study, from which it would seem smoking is not implicated, ¹⁶ for, as we can see (fig 3a), there is a fourfold difference in rates between the United Kingdom and Japan for a similar level of tobacco consumption. ^{17 18} However, if we make a within cultural comparison looking just at Western countries there is the dose-response relation that would be expected. ¹⁹

So, returning to the example of heart disease from figure 2, the logic of our findings for smoking and lung cancer must be that the cross cultural comparison between Japan and Finland in favour of diet being a major causative factor is much less secure than the evidence against from the within cultural comparison of Switzerland and Finland.

Holmes: Proceed, Watson, I am all ears.

Watson: When we turn to the migrant studies I again found the same difficulty that the relevant data are simply not cited in the medical literature. There are numerous articles investigating the disease rates of Japanese migrants to the United States, but it was only with considerable determination that I finally found the data you requested. It is not, as you will by now have anticipated, good news. Consider the Swedes. The proportion of fat in their diet is similar to that of the citizens of the United States, even though their rates of heart disease are considerably lower (300 per 100 000 for men aged 35-64). What do you think happens when they move to the United States? Their dietary practices may remain much the same, but their heart disease rates shoot up to that found in their adopted country (572 per 100 000). ²⁰

Indeed, it would seem to be a generalised phenomenon that migrants will exchange the entire pattern of disease of their home country for that of their adopted country. So, even without the compelling contrary evidence of the Swedes, it would still not be permissible to cite the rates of heart disease of migrants as evidence for a specific causal relation between diet and disease.

Holmes: Capital, Watson, capital.

Watson: From all this I can only conclude that the protagonists have misled themselves by concentrating only on those aspects of the circumstantial evidence that would seem to substantiate their case, while ignoring that which challenges it.

Holmes: I fear, Watson, you are characteristically being too generous. I have, to your irritation, already referred on a couple of occasions to the case involving the famous racehorse Silver Blaze, which disappeared on the eve of the Wessex Cupa race he was tipped to win on the same night that the body of his unfortunate trainer, John Straker, was found not far from the stable, his head having been shattered by a strong blow. The circumstantial evidence pointed strongly, indeed convincingly, to the young man Fitzroy Simpson as the perpetrator of this crime, but I, as you will no doubt remember, was unconvinced.

Watson: Quite so. The reason you suspected Mr Straker might be the cause of his own misfortune was…

Holmes: The curious incident of the dog in the night time. And what was that curious incident? The dog guarding Silver Blaze did not bark, presumably because the abductor must have been known to him most probably his master. And who was his master? Why, none other than the horse’s trainer, John Straker. And then, when Straker sought, for his own devious reasons, to hobble Silver Blaze, the horse lashed out and killed him with a blow to the head before disappearing into the night.

Watson: Holmes, I think I get your drift. Those who were most familiar with the circumstantial evidence were alone in a position to know its inconsistencies the inconsistencies I have spent so long ferreting out but they did not bark.

Holmes: Precisely. And when the decline in rates of heart disease became so dramatic as to make it highly improbable that it could be attributed to changes in the putative risk factors, once again they did not bark. The important findings were in an impenetrable list of figures including the "average annual percentage change of coronary event rates over the last five years."

You know, I think we should call this "The case of the missing data" because the data for every contrary strand of evidence were missing, or at least difficult to find. As my legal friends would put it, "details that could reasonably have been expected to appear in evidential text were absent."

Watson: So, heart disease remains an enigma though the striking rise and fall over the past 50 years is strongly suggestive of a biological cause. No doubt those who smoke or take insufficient exercise or whose cholesterol concentrations are greatly raised may be at "increased risk," but none can be determinant (in the way the putative biological cause clearly must be), which is why the pattern of the disease has changed so dramatically quite independently of them. I can hardly wait to smell once again the aroma of a cooked breakfast with an easy conscience.

Holmes: Watson, your wish will be granted, and I will instruct Mrs Hudson accordingly. Meanwhile, given everything we have learnt today and how fascinating it has been we should perhaps usefully turn our attention to investigating why your fellow doctors have been persuaded to prescribe cholesterol lowering drugs on so massive a scale. ^{21 22} But that is for another day.