I show you a coin, and I tell you “both sides of this coin are heads”. However, I refuse to give you the coin—for whatever reason, I insist on demonstrating this fact by repeatedly flipping it. If I’m telling the truth, it’ll always come up heads.

Okay, so I start flipping it, and lo and behold it always comes up heads. I flip it, say, \(1000\) times, and every time it comes up heads. The chance of this happening with a fair coin is \(2^{-1000}\). Is your probability that the coin is fair now of order \(2^{-1000}\)?

No, of course not. The coin could have two heads (maybe it’s more likely than not), but it could also be that I’m somehow being sneaky in flipping it. Maybe it’s weighted (probability at least a few percent), or I’m playing some trick with a magnet (seems like a good idea; probability at least 1%), or I’ve drugged you into hallucinating. All of these probabilities are comically larger than that naive \(2^{-1000}\). And none of these probabilities is affected by you watching me flip the coin another time!

This is part of why constructing small probabilities is hard. It is never sufficient to keep gathering more of one type of evidence, since eventually you run up against the (unmoving) probability that this type of evidence happens to be worthless. Your posterior probability estimate saturates at some value strictly less than \(1\) (or greater than \(0\)).

Hopefully this is all obvious, but it has some consequences.

Alex Tabarrok argues that one should trust literature, not papers. I cannot agree. A paper that makes a concrete argument, I can read closely, follow, and come to a sort of “meeting of minds” with the author. In other words I can understand it, and whatever pieces of evidence the authors believe they have found, I can take them on board as evidence myself. If it’s a good paper, then at the end, I will have updated my estimates in the same manner as the authors did. A paper—one paper—is a fantastic tool for this. I know of no better.

A literature, though. I cannot read and understand a literature. The existence (and size, and other bulk characteristics) of the literature must themselves serve as evidence. But this is poor evidence! If a priori I believe that “a literature on a topic this important has a 40% chance of being substantially corrupted by social/political/economic forces”, then the maximum probability I can assign to a typical claim is going to be 60%. (Those are, in fact, typical of my estimates in these situations.)

This wouldn’t be such a problem if I knew how to evaluate a large set of papers to determine whether there are such corrupting forces at work. But I don’t—certainly not well enough to reduce that 40% below, say, 20%. I have enough trouble constructing probabilities outside of \([0.1,0.9]\) in physics, a field without strong external social forces (and where I actually work).

The result is that, outside of cases where a single paper (or a small set of papers) can lay out a convincing argument, a rational person probably shouldn’t be strongly convinced of any individual claim coming out of research. Scenarios where the literature is systemically biased are too likely, and are not made much less likely (at least as far as I can tell) by the literature being large, or using diverse methods, or using the latest techniques.