Pass / Fail Mentality

(Following on from The Calibration Problem: Why Science Is Not Deductive)

Recently, I was talking about calibration (here and here), and how it should be more than just identifying the most likely cause of the output of a measuring instrument. The calibration process should strive to characterize the range of true conditions that might produce such an output, along with any major asymmetries (bias) in the relationship between the truth and the instrument's reading. In short, we need to identify all the major characteristics of the probability distribution over states of the world, given the condition of our measuring device.

Failure to go beyond simply identifying each possible instrument reading with a single most probable cause is a special case of a very general problem that in my opinion plagues scientific practice. Such a major failure mode should have a special name, so lets call it 'pass / fail mentality.' It is the most extreme possible form of a fallacy known as spurious precision, and involves needlessly throwing away information.

Signal and Noise

Here's a useful thought experiment (slightly reworded) from Ronald Fisher's well-known text book from 1925, 'Statistical methods for research workers':

In each of two nearly identical universes, agricultural researchers wanted to compare 2 fertilizers (I know, it sounds like bullshit). In each universe, similar protocols were performed (this really happened, I swear): 2 plots of land were each divided into two parts, and the different parts treated with the different fertilizers. The same crop plant was cultivated on each part of each plot, and the individual yields recorded. The yields, in tons, were:

Universe 1:

plot	fertilizer A	fertilizer B
1	20	28
2	23	32

Universe 2:

plot	fertilizer A	fertilizer B
1	20	28
2	23	41

In which universe is there stronger evidence for the advantage of fertilizer B over fertilizer A?

In universe 1, the average advantage is 8.5 tons per plot, while in universe 2, the advantage is 13 tons. Seems like the farmers in universe 2 can place greater justified faith in fertilizer B.

But that conclusion is a bit too quick. There's more to the strength of evidence than just the magnitudes of the averages. We also have to consider the quality of the evidence, the signal to noise ratio. In universe 1, the numbers for each fertilizer are tightly clustered, (20 is not much different from 23, and 28 is not much different from 32) supporting the idea that the experiment was well controlled - random factors probably contributed little to the outcomes. 

In universe 2, however, the experiment doesn't look as well controlled. There's a big difference between the 2 results for fertilizer B, indicating that there is much more going on than just the choice of fertilizer. There's apparently more noise in this case, and if there's that much noise, then maybe the outcome of the experiment is purely down to random chance.

To analyze the relationship between the signal and the noise, Fisher recommends a null-hypothesis significance test (well, he invented significance tests, after all). Modelling the two sets of samples (A and B) as drawn from the same normally distributed population (the null hypothesis), we can calculate the plausibility of the observed difference between their means under the null hypothesis, H₀. If the observed data, D, is too implausible under H₀, then, as tradition goes, H₀ is rejected. The problem is, we only have a few samples from which to calculate the width of that normal distribution, so another distribution, Student's t-distribution, which accounts for the uncertainty of the standard deviation, is used instead. To get a p-value, we have to calculate a t-statistic, and integrate the t-distribution from that statistic out to infinity to obtain the desired implausibility of D under H₀.

To get the t-statistic, we can first calculate an aggregate sample standard deviation, s:

where x_i, m_i, and n_i are respectively yields, averages, and the numbers of samples, for fertilizer i.

The t-statistic for comparison of two means (where H₀ states that the 2 means are the same) is then given by

Taking account of the number of degrees of freedom in the experiment, n_A + n_B - 2, equal to 2, tables or almost any mathematical software are consulted to perform the required integral, which gives directly the p-value. For universe 1, the p-value I get from this test (two-tailed integration) is 0.041 - quite significant, the null hypothesis is on shaky ground. For universe 2, however, where we noticed that there was apparently a far greater random component to the data, the p-value is 0.11. This is almost 3 times larger, meaning that the data are here more believable under H₀. We have weaker grounds for supposing that the fertilizers perform any different in universe 2.

This rare foray into the realm of orthodox stats has been hopefully sufficient to illustrate the point about quality of information, in terms of signal and noise (don't ask for guarantees that I did everything correctly, I may never understand the mindset under which these significance tests make sense). What I don't like, though, about the null-hypothesis significance test (among other things) is that no alternate hypotheses are formulated or evaluated. Without comparing H₀ to any other H's, the whole process is frankly rather hollow. What's more, if H₀ is rejected, then further machinery is required to figure out how large the effect is. What I want, generally, is a set of proper probabilities, worked out for a whole range of possibilities, including H₀.

Just for a laugh, then, I'll work through an approximate model, that'll allow me to plot a continuous probability distribution for a whole range of values for the average difference between the yields for the 2 fertilizers. I'll use the t-statistic again, but I won't be integrating tails (at least, not until after I have a posterior distribution). This t-statistic will help me get around the ambiguity concerning the width of the noise distribution, when calculating the likelihood function.

We can test the significance of the mean of a single set of samples from a single population, relative to some hypothetical mean, μ₀, using another formula for the t-statistic:

where s is the regular sample standard deviation. (For emphasis, the reason for the different formula is that we are doing a different test - looking at a single mean, <x>, as opposed to comparing 2 means.)

The likelihood function, P(D | HI),  is calculated by evaluating the t-distribution with this statistic and the number of degrees of freedom, n - 1, which is 1.

The single population we're looking at is the population of differences between fertilizer B and fertilizer A. The parameter to vary in order to generate the likelihood function is the hypothesis, μ₀.

As a prior density, I'll use a normal distribution centered at zero. To assign a width, I'll set the standard deviation to 15 tons, which means that there is a very small probability (< 5%) that the absolute value of the difference in yields for the two fertilizers exceeds 30 tons. The posterior probability is then simply the normalized product of this prior and the likelihood function, from Bayes' Theorem.

The graph below shows the posterior probability density as a function of μ_B-A, for each of our universes. The curves illustrate clearly the impact of decreasing the signal to noise ratio in universe 2: though the peak is further to the right, the tail extends further to the left, due to the lower sensitivity of the experiment in that universe. Integrating the two curves from -∞ to 0, we see that the probability that fertilizer B is actually not better than fertilizer A is twice as large in universe 2 as it is in universe 1, which is similar to the result above, in terms of p-values. As the old saying goes: garbage in, garbage out. Precise inference demands a well-controlled experiment.

Its only human that quite often in the quest for knowledge we'll derive greater confidence from results like universe 2, rather than universe 1. It takes care not to be seduced by a greater overall difference, before taking time to consider how much of that difference is likely to be due to random fluctuations. Often, for brevity, we'll summarize an experiment by recording only the mean result (or in less formal circumstances subconsciously estimate the mean, and forget all the other details), but as we've seen, to draw good-quality inferences we need to note not only the mean but also the dispersion and the number of samples (roughly, confidence in a result scales¹ according to SNR × √ n  ). How many figures quoted by politicians or newspapers (or anybody else with influence) lose their sting when we notice that no error bar has been provided? Context is all important. Sometimes, only moderately careful analysis is enough to overturn an intuitively appealing conclusion, and it's results like this that show the importance of a cultivated awareness of the mathematical machinery of rational inference.

---

[1]	'Why randomized controlled trials fail but needn't: 2. Failure to employ physiological statistics, or the only formula a clinician-trialist is ever likely to need (or understand!),' D.L. Sackett, CMAJ October 30, 2001 vol. 165 no. 9, link

The Higgs Boson at 5 Sigmas

Huge congratulations to Peter Higgs and the other theorists who, half a century ago, predicted this particle as part of the standard model, and to the experimentalists at CERN involved in making this recent discovery. We now have extremely compelling evidence of a new particle (new to us, at least) consistent with the Higgs boson. 

Sadly, there's much confusion circulating about the meaning of the 5σ significance level reported for the data on the Higgs. Lets first clear up what this means, before examining some of the confusion.

The null hypothesis, which states that there is no signal present in the data, only noise, implies that the magnitude for some parameter, Y, is some value, μ. Since there is random noise in the data, however, the value of Y observed in a measurement is unlikely to be exactly μ, but will follow a probability distribution with mean μ, and standard deviation σ. If the null hypothesis is true, then we can expect a measurement of Y to produce a result close to μ. The probability is less than 6 × 10^-7 that a measured value will be 5 standard deviations or further from μ, assuming the null hypothesis is true. This number is the p-value associated with the measurement. (In fact, for this case, a single-tailed test was performed, so the reported p-value for the Higgs boson is half this, about 2.8 × 10^-7.)

To recap, the reported p-value is the probability to observe data as extreme as or more extreme than than the data observed, assuming the null hypothesis is true. Roughly speeking, the p-value is P(D | H₀). David Spiegelhalter, over at the Understanding Uncertainty blog, has been monitoring press reports on the recent Higgs announcement, finding a high degree of misunderstanding among the journalists. Many writers described the reported p-value as the probability that the null hypothesis is true, calculated from the data, but this is P(H₀ | D), a very different thing, as I have described before.

Spiegelhalter has found numerous examples of this error, including writers from New Scientist and Nature, who really ought to know better. These are presumably professional journalists, however, who can perhaps be granted some slack, but what about a renowned cosmologist? How about these words:

Each experiment quotes a likelihood of very close to “5 sigma,” meaning the likelihood that the events were produced by chance is less than one in 3.5 million. 

These words come from Lawrence Krauss, a highly respected theoretical physicist, and they are wrong. I mentioned in my very first blog post that physicists are not generally so great at statistics, but I really don't like being proven right so blatantly. I suppose we really mustn't feel too bad about the mistakes in the press, when the experts can't get it right either.

In the article linked above, Spiegelhalter praises a couple of writers for getting the meaning of the p-value the right way round, but really these authors also fail to properly understand the matter. The passages, respectively from the BBC and from the Wall Street Journal, were:

...they had attained a confidence level just at the "five-sigma" point - about a one-in-3.5 million chance that the signal they see would appear if there were no Higgs particle.

and

If the particle doesn't exist, one in 3.5 million is the chance an experiment just like the one announced this week would nevertheless come up with a result appearing to confirm it does exist. 

While both these passages deserve praise for recognizing the p-value as the probability for the data, rather than the probability for the null hypothesis, they unfortunately are still not correct, for a reason that will bring me neatly in a moment to my second major point about the standard of reporting results such as these in terms of p-values. The reason these passages are wrong is that they employ an overly restrictive interpretation of the null hypothesis. They assume H₀ is the same as '"the Higgs boson does not exist," whereas the true meaning is broader: "there is no systematic cause of any patterns present in the data." If we reject the null hypothesis, we still have other possible non-Higgs explanations to rule out before we are sure that the Higgs boson is not a fantasy. These alternatives include other particles, consistent with different physical models; systematic errors in the measurements; and scientific fraud.

This might seem like a subtlety not really worth complaining about, but along with the other errors mentioned, it is a perpetuation of the fallacies surrounding the frequentist hypothesis tests - fallacies that mask the fact that the p-value is a poor summary of a data set, as I have explained in The Insignificance of Significance Tests. Why, for example, is it considered so important to rule out the null hypothesis so comprehensively, when there is little or no attempt to quantify the other alternative hypotheses? The possibility that the observation is a new particle, but not the Higgs, will no doubt be investigated substantially, but presumably will never be presented as a probability. The probability for fraud or systematic error will presumably receive absolutely no formal quantification at all.

Recently, we had a good example of how calculating the probability for measurement error could have prevented substantial trouble. As Ted Bunn has explained, it was a simple matter to estimate this quantity when researchers publicized results suggesting superluminal velocities for neutrinos, yet several groups set up experiments to try to replicate those result, at significant expense, despite the obviousness of the conclusion.

The traditional hypothesis tests set out to quantify, in a rather backwards way, the degree of belief we should have in the null hypothesis, but this is not what we are really interested in. What we want to know when a we look at the results of a scientific study is: what is the probability that the hypothesized phenomenon is real? For this, we should calculate a posterior probability. To do this, we need a specified hypothesis space with an accompanying prior probability distribution. This is one of the things that leads to great discomfort among some statistical thinkers, and was instrumental in leading to the adoption of the p-value as the standard measure of an experiment: how can you obtain a porsterior probability without violating scientific objectivity? How can you assign a prior probability without introducing unacceptable bias into the interpretation of the observations?

The people who think like this, though, don't seem to realize that prior probabilities can be assigned in a perfectly rigorous mathematical way, that will not introduce any unwarranted, subjective degree of belief. There seems to be a feeling like 'we don't know enough to specify the prior probability correctly,' but this is ridiculous - missing knowledge is exactly what probability theory is for. Any probability is just a formally derived, distilled summary of existing information. To calculate the probability for the Higgs, in principle all you need to do is start with a non-informative prior (e.g. uniform), figure out all the relevant information, and bit by bit account for each piece of information using Bayes' theorem. Sure, the bit about figuring out the relevant information is hard, but all those clever particle physicists at CERN must be able, if they put their minds to it. Then it is just a matter of investigating how well the various hypotheses, with and without the Higgs boson, account for various observations, and how efficiently they do so. We'll get a glimpse of how this is done, in a future post, when I get around to discussing model comparison, and a formal implementation of Ockham's razor.

Yes, there will always need to be assumptions made, in order to carry out such a program, but inductive inference is impossible without assumptions, (if you don't believe me, try it!) and any competent scientist (almost by definition) will be able to keep the assumptions used to those that either are accepted by practically everybody, or make negligible difference to the result of the calculation. It is the frequentists who are committing a fallacy by trying to learn in a vacuum.

I find it a great pity that the posterior distribution was not the chosen route taken for reduction of the data from the Large Hadron Collider. Firstly, this is an extremely fundamental question, and if any question deserves the best possible answer from the available data, this is it. The p-value simply doesn't extract all the information from the observations we now have at our disposal. Secondly, yes: it is a hugely complicated problem to assign a prior probability to a hypothesis like this, but this is a project involving a huge number of presumably some of the finest scientific minds around - if anybody can do it, they can. If they did, it would first of all prove that it can be done, and secondly it would lay a very strong foundation for the development of general techniques and standards for the computation of all manner of difficult-to-obtain priors.

The Raven Paradox

Earlier, in 'The insignificance of significance tests', I wrote about the silliness of the P-value, one of the main metrics scientists use to determine whether or not they have made a discovery. I explained my view that a far better description of our state of knowledge is the posterior probability. C.R. Weinberg, however, presumably does not agree with all my arguments, having written a commentary some years ago with the title 'It's time to rehabilitate the P-value.'¹ Weinberg acknowledges the usefulness of the posterior probability in this article, and I agree with various other parts of it, but a major part of her argument in favour of P-values is the following statement, from the start of the article's second paragraph:

'The P-value quantifies the discrepancy between a given data set and the null hypothesis,'

Lets take a moment to consider whether or not this is true.

Firstly, lets note that the word 'discrepancy' has a precise meaning and an imprecise one. The discrepancy between 5 and 7 is 2, and has a precise formal meaning. What is the discrepancy, though between an apple and an orange? When we say something like 'there is a discrepancy between what he said and what he did,' this makes use of the general, imprecise meaning of 'disagreement'. What is the discrepancy between a data set and a hypothesis? These are different things and can not be measured against one another, so there is clearly no exact formal meaning of the discrepancy in this case. It is a pity then that the main case made for use of P-values, in this case, is based on an informal concept, rather than an exact idea, which scientists normally prefer.

So it is clear then, that the discrepancy between a data set and the null hypothesis has no useful, clearly definable scientific meaning (and, therefore, can not be quantified, as was the claim). Lets examine, though, what the term 'discrepancy' is trying to convey in this case. Presumably, we are talking about the extent to which the data set alters our belief in the null hypothesis - the degree of inconsistency between the null hypothesis and the observed facts. Surely, we are not talking about any effects on our belief in the data, as the data is (to a fair approximation ²) an empirical fact. Anyway, belief in the data was not the issue that the experiment was performed to resolve, right? The truth or falsity of a particular data set is a very trivial thing, is it not, next to the truth or falsity of some general principle?

But the P-value does not quantify our rational belief in any hypothesis. This is impossible. For starters, as I hinted at in that earlier article, by performing the tail integrals required to calculate P-values, one commits an unnecessary and gratuitous violation of the likelihood principle. This simple (and obvious) principle states that when evaluating rational changes to our belief, one uses only the data that are available. By evaluating tail integrals, one effectively integrates over a host of hypothetical data sets, that have never actually been observed. 

Even if we eliminate the tail integration, however, and calculate only P(D | H₀), we are still not much closer to evaluating the information content of our data, D, with respect to H₀. To do this, we need to specify a set of hypotheses against which H₀ is to be tested, and obtain the direct probability, P(D | H), for all hypotheses involved. To think that you can test a hypothesis without considering alternatives is, as I have mentioned once or twice in other articles, the base-rate fallacy.

One of my favorite illustrations of this basic principle, that the probability for a hypothesis has no meaning when divorced from the context of a set of competing hypotheses, is something called the raven paradox. Devised in 1945 by C.G. Hempel, this supposed paradox (in truth, there seem to be no real paradoxes in probability theory) makes use of two simple premises:

(1) An instance of a hypothesis is evidence for that hypothesis.

(2) The statement 'all ravens are black' is logically equivalent to the statement 'everything that is not black is not a raven.'

Taking these two together, it follows that observation of a blue teapot constitutes further evidence for the hypothesis that all ravens are black. While it seems to most that the colour of an observed teapot can convey little information about ravens, this conclusion is claimed to be logically deduced. This paradox got several philosophers tied up in knots, but it took a mathematician and computer scientist, I.J. Good (he worked with Turing both at Bletchley Park and in Manchester, making him one of the world's first ever computer scientists) to point out its resolution.

The obvious solution is that premise number (1) is just plain wrong. You can not evaluate evidence for a hypothesis by considering only that hypothesis. It is necessary to specify all the premises you wish to consider, and specify them in a quantifiable way. Quantifiable, here, means in a manner that allows P(D | H) to be evaluated. To drive this point home, Good provided a thought experiment (a sanity check, as I like to it) where, given the provided background information, we are forced to conclude that observation of a black raven lowers the expectation that all ravens are black.³

Good imagined that we have very strong reasons to suppose that we live in one of only two possible classes of universe, U₁ and U₂, defined as follows:

U₁ ≡ Exactly 100 ravens exist, all black. There are 1 million other birds.

U₂ ≡1,000 black ravens exist, as well as 1 white raven, and 1 million other birds.

Now we can see that if a bird is selected randomly from the population of all birds, and found to be a black raven, then we are most probably in U₂. From Bayes' theorem:

P(U1 | DI)  =   P(U1 | I) × P(D | U1 I) P(U1 | I) × P(D | U1 I) + P(U2 | I) × P(D | U2 I)

No information has been supplied concerning the relative likelihoods of these universes, so symmetry requires us to give them equal prior probabilites, so

P(U1 | DI)  ≈   10-4 10-4 + 10-3

which makes it about 10 times more likely that we are in U₂, with some non-black ravens.

---

[1] C.R. Weinberg, 'It's time to rehabilitate the P-value,' Epidemiology May 2001, Vol. 12 No. 3 (p. 288) (Available here)

[2] We can imagine the following conversation between a PhD student and his advisor:

Supervisor:

This new data is fantastic! And so different to anything we have seen before. Are you sure everything was normal in the lab when you measured these?

Student:

Well, I was a little bit drunk that day....

But this kind of thing is rare, right? Come on, back me up here guys.


[3] I.J. Good, 'The White Shoe is a Red Herring,' British Journal for the Philosophy of Science, 1967, Vol. 17, No. 4 (p. 322) (I haven't read this paper (paywall again), but I have it from several reliable sources that this is where the thought experiment, or an equivalent version of it, appears.)

The Insignificance of Significance Tests

In an earlier post on the base-rate fallacy, I made use of two important terms, ‘false-positive rate’ and ‘false-negative rate,’ without taking much time to explain them. These are concepts we need to be careful with, because, simple though they are, they have been given terrible names.

Lets start with the false positive rate for a test. This could mean any one of a number of things:

(1) it could be the expected proportion of results produced by the test that are both false and positive 

(2) it could be the proportion of positive results that are false 

(3) it could be the proportion of false results that are positive

So which one is it? Are you ready?

None of the above.

The false positive rate is the proportion of negative cases that are registered as positive by the test. This definition is widespread and agrees with the Wikipedia article ‘Type I and Type II Errors’. If it is a diagnostic test for some disease, it is the fraction of healthy people who will be told that they have the disease (or referred for further diagnosis).

My claim that the term is confusing, however, is supported by looking at another Wikipedia article, ‘False Positive Rate,’ which provides the definition: “the probability of falsely rejecting the null hypothesis.” The null hypothesis is the proposition that there is no effect to measure – what I have called a negative case. The probability of falsely rejecting the null hypothesis, therefore, depends on the probability that there is no effect to measure, while the proportion of negative cases that are registered as positive does not. The alternate definition in this second Wikipedia article is the same as my number (1), above.

Number (2) on the list above is the posterior probability that the null hypothesis is true, given that the test has indicated it to be false. It is obtained using Bayes’ theorem. Confusion between this posterior probability and the false-positive rate is the base-rate fallacy, yet again. Unfortunately, there is little about the term ‘false-positive rate’ that strives to steer one away from these misconceptions.

What makes the situation much worse is that most scientific research is assessed using the false-positive rate, while what we should really be interested in is assigning a posterior probability to the proposition under investigation.

The false-positive rate is often denoted by the greek letter α. In classical significance testing, α is used to define a significance level – or rather the significance level defines α. The significance level is chosen such that the probability that a negative case triggers the alarm is α. A negative case, once again, is an instance where the null hypothesis is true, and there is no effect. For example, if two groups are being treated for some affliction, one group with an experimental drug and the other with a placebo, the null hypothesis might be that both groups recover at the same rate, as the new treatment has no specific impact on the disease. If the null hypothesis is correct, then in an ideal measurement, there will be no difference between the recovery times of the two groups. 

But measurements are not ideal: there is ‘noise’ – random fluctuations in the system under study that produce a non-zero difference between the groups. If we can assign a probability distribution for this noise, however, we can define limits, which, if exceeded by the measurement, suggest that the null hypothesis is false. If x is the measured difference in recovery time for the 2 groups, then there are two points, x_α/2, on the tails of the distribution, such that the integral of one of the tails up to this limit is α/2. The total probability contained in these two tail areas, therefore, sums to α. The x_α/2 points are chosen so that α is equal to some desired significance level, some acceptably low false positive rate. (We can not make the false positive rate too small, because then we would too often fail to spot a real effect – the false negative rate would be too high.) The integration is performed on each side of the error distribution, as we have no certainty in which direction the alternate hypothesis operates: the recovery time with the new drug might be worse than with no treatment, which could still lead to rejection of the null hypothesis.

The null hypothesis predicts a value of x in the peak of the curve, but measurement noise leads to a probability distribution for the experiment as plotted. The x_α/2 points are chosen so that the tails areas either side (the shaded regions) sum to 5%, or whatever the desired significance level is. In classical hypothesis testing, any measured x further from the centre than either of these points is considered statistically significant.

A common value chosen for α is 0.05. That is, if the null hypothesis is true, then the measured value of x will exceed x_α/2 about one time in twenty. This is the basis of how results are reported in probably most of science – if x exceeds the prescribed significance level, then the null hypothesis is rejected, and the result is classified as statistically significant and considered a finding. If the measured value of x is between the two x_α/2 points, then the null hypothesis is not rejected, and the outcome of the study is probably never reported (a fact that contributes hugely to the problem of publication bias in the scientific literature). This system for interpreting scientific data and reporting outcomes of research is, lets be honest, a travesty.

Firstly, the whole rationale of the significance test seems to me to be deeply flawed. The idea is that there is some cutoff, beyond which we can declare the matter final: ‘Yes, we have a finding here. Grand. Nothing more to do on that question.’ How manifestly absurd? Such a binary declaration about a hypothesis, necessary if one wishes to take real-world action based on scientific research, needs to be based on decision theory, which combines both probability theory and some appropriate loss function (something that specifies the cost of making a wrong decision). But the declarations arising in decision theory are not of the form ‘A is true,’ but rather ‘we must act, and rationality demands that we act as if A is true.’ Using some standard α-level is about the crudest substitute for decision theory you could imagine. 

Why should our test be biased so much in favour of the null hypothesis, anyway? The alternate hypothesis, H_A, almost always represents an infinity of hypotheses about the magnitude of the possible non-null effect, so a truly 'unbiased' starting point would seem to be one that de-emphasizes H₀. Remember that at the core of the frequentist philosophy (not the approach I wish to promote) is the dictum "let the data speak for themselves": don't let prior impressions taint the measurement.

Secondly, when a finding is reported with a false-positive probability of 0.05, there appears to be a feeling of satisfaction among the scientific community that 0.05 is the probability that the positive finding is false. But meta-analyses regularly do their best to dispel this myth. For example, looking only at genetic association studies, Ioannidis et al.¹ reported that 'results of the first study correlate only modestly with subsequent research on the same association,' while Hirschorn et al.² write that 'of the 166 putative associations which have been studied three or more times, only 6 have been consistently replicated.'

Thinking again about the method for calculating the positive predictive value for a diagnostic test, the probability that a positive finding is false is not determined only by the false-positive rate, α, but also by the false negative rate, and the prior probability. The false negative rate, by analogy with the false positive rate, is the proportion of real effects that will be registered as null findings. It depends on the number of samples taken in the study and the magnitude of the effect. In the medical trial example, if the new drug helps people recover twice as fast, then there will be fewer false negatives than if the difference is only 10%, and a trial with 100 patients in each group will give a more powerful test than a trial with only 10 in each group.

If D_f represents data corresponding to a finding, with x > x_α/2, then from Bayes’ theorem, the probability that the null hypothesis will have been incorrectly rejected is

P(H0 | Df, I)  =   P(H0 | I)P(Df | H0, I) P(H0 | I)P(Df | H0, I) + P(HA | I)P(Df | HA, I)

(1)

P(D_f | H₀, I) is α, and P(D_f | H_A, I) is 1-β, where β is the false negative rate. Using the standard α level of 0.05, I plot below this posterior probability as a function of the prior probability for the alternate hypothesis, H_A, for several false-negative rates. To estimate the posterior error probability for a specific experiment, Wacholder et al.³ propose to use the p-value instead of α (the p-value is twice the tail integral up to x_m, where x_m is the measured value of x, rather than x_α/2). Strictly, one shouldn't use the p-value, but P(x_m | H₀, I). We use α to evaluate the method and so integrate over all possible outcomes that would be registered as findings, while we use P(x_m) to investigate a particular experiment – there is only one outcome, so no integration is required. I’m fairly sure Wacholder et al. are aware of this: their goal seems to be to provide an approximate methodology capable of salvaging something useful from an almost ubiquitous and highly flawed set of statistical practices. In this regard, I think they can probably be credited with having made a valuable contribution. The problem with this, though, is that for a specific data set, P(D | H_A, I) is not the same as 1-β, and can not be determined. The correct procedure, of course, requires resolving H_A into a set of quantifiable hypotheses and calculating the appropriate sampling distributions for each of them.

Posterior probability for H₀ following a statistically significant result, with α set at 0.05, plotted vs the prior probability that H₀ is false.

We can see readily from the above plot that the posterior probability varies hugely, even for a fixed α, and that α alone (or the p-value) is next to useless for predicting it. As the prior probability gets smaller, the posterior error probability associated with a positive finding approaches unity. Looking closely at a prior of 0.01, which is generous for many experiments, (especially, for example, large-scale genomics studies, where the data and its analysis are now cheap enough to permit almost every conceivable relationship to be tested, regardless of whether or not they are suggested by other known facts) we can see that for a low-power test, with β = 0.8, then P(H₀ | D_f, I) is over 96%. So we crank up the number of samples, improve our instruments, do everything we can to reduce the experimental noise, until, miracle of miracles, we have reduced the false negative rate to almost zero. What is P(H₀ | D_f, I) now? Still 83%. Bugger.

This is not to say that the experiments are not worth doing. Science evidently makes tremendous advances despite these difficulties, and the technology that follows from the science is the most obvious proof of this. (Besides, any substantial success in reducing β will also permit an associated reduction of α.) What it does mean, however, is that the standard ways of reporting ‘findings,’ with alphas and p-values, are desperately inadequate. They fail to represent the information that a data set contains and to convey what should be our rational degree of belief in a hypothesis, given the empirical evidence available. Evaluation of this information content and elucidation of these rational degrees of belief (probabilities) should be the goal of every scientist, and the communication of these things should be viewed as a privilege to take delight in.

---

Update (19-5-2012)

Previously I stated that:

'Colhoun et al.⁴ found that 95% of reported findings concerning the genetic causes of disease are subsequently found to be false.'

This was based on a statement by Wacholder et al.³ that:

'Colhoun et al. estimated the fraction of false-positive findings in studies of association between a genetic variant and a disease to be at least .95.'

I have subsequently checked this paper by Colhoun et al., and could not find this estimate. I have adjusted the text accordingly, adding new references that support my original point. I apologize for the error. My only excuse is that, locked as it was behind a paywall, I was unable to access this paper for fact checking, before making a special trip to my local university library. I still recommend the Colhoun et al. paper for their discussion of the unsatisfactory nature of evidence evaluation in their field.

---

[1] J.P. Ioannidis et al. ‘Replication validity of genetic association studies,’ Nature Genetics 2001, 29 (p. 306)

[2] J.N. Hirschorn et al. ‘A comprehensive review of genetic association studies,’ Genetics in Medicine 2002, 4 (p. 45)  Available here.

[3] S. Wacholder, et al. ‘Assessing the probability that a positive report is false: an approach for molecular epidemiology studies,’ Journal of the National Cancer Institute, Vol. 96, No. 6 (p. 434), March 17, 2004. Available here.

[4] H.M. Colhoun et al. ‘Problems of reporting genetic associations with complex outcomes,’ Lancet 2003 361:865–72