We published two findings and corrected one. A one-line check told them apart before we knew

The two findings, on the day we published them

The half-life median sat at 2 months and stayed at 2 months no matter which of the 16 titles was dropped. Nothing in the sample was holding it up. The floor-change median could be pushed anywhere between -11.8% and -5.8% by removing one of five. We reported it anyway.

What happened next

We widened the set from 35 titles to 59 for an unrelated reason. Here is where the same two medians landed.

One held to the number. The other fell to a third of what we had published, and we corrected the article that carried it — the second correction on that piece that day. The check would have cost one line and it would have told us which was which.

The check we would normally have reached for does not work here

The interquartile range is the usual way to describe spread, and on these two samples it says they are almost the same.

The interquartile range cannot do this job. On the five values behind our first publication it read 1.78 times the median, which looks narrow — because it puts the one extreme value outside its own range and stops looking at it. Leave-one-out asks what happens when that value is the one removed.

The five values behind that first publication were −27.8, −16.9, −6.7, −5.0 and +60.2. The interquartile range is computed from the middle of that list and never has to look at +60.2 at all. Leave-one-out asks what the answer becomes when +60.2 is the one taken away, which is exactly the question a small sample raises.

How to run it

Take the numbers you have. Remove one, recompute the median. Do that once for every number. If the median barely moves, one more or one fewer observation will not change your finding. If it moves by a large fraction of itself, it will.

There is no sampling and no random seed here. The same numbers always give the same answer, which is why we can put it in a build script. We call it leave one out rather than by its statistical name because the name is the whole method. Our threshold is 0.5 — a median that moves by half its own size when one observation leaves is not yet a finding.

What this page cannot tell you

Leave-one-out does not tell you a finding is wrong. It tells you the sample is not yet large enough for that finding to be reported as one. Those are different things and we have conflated them before.

It works on a median. A share, a total or a correlation needs a different check, and we do not have one for those yet.

A steady median is not a true one. Every title here was chosen by us, and a biased sample can give a very steady wrong answer — this check cannot see that at all.

Two findings is not a study of findings. We are describing what we did today, not establishing how often this happens.