A common sentiment is that trained scientists have an important skill, often called “taste”, which is crucial in identifying worthwhile work. For a recent example see Tim Hwang and Caleb Watney’s most recent piece at Macroscience, which summarizes this view pretty well:

Scientists will often scoff at traditional metrics like citations or patents because, in their view, good science is a “know-it-when-you-see-it” phenomenon and fundamentally about good taste. Large-scale surveys of scientists could help bridge this gap by aggregating the opinions of scientists about promising (or unpromising) developments in their field and in adjacent fields.

If we’re going to assert that thus-and-such mechanism is an effective one, it’s good to have a story for what causes the mechanism to work. In this case the most plausible story looks something like this:

1. Scientists are repeatedly exposed to examples of historical work in their field and adjacent fields.
2. Thanks to obvious selection biases, that work is disproportionately the high-impact work.
3. Scientists begin to develop a feel for what sort of work is “high-impact”.
4. Work that feels familiar (and thus similar to historical high-impact work) is held to be in good taste.

Note that the selection bias in step 2 is doing heavy lifting here—and it’s good heavy lifting. It’s nice to see selection bias being the good guy for once!

Unfortunately, this procedure breaks down for the sort of work that spawns a new (sub)field. By definition and perhaps design, that work doesn’t look much like historically high-impact work. “Taste”, when driven by the mechanism above, is only sensible within the field that spawns it. This isn’t so surprising—who would expect a particle theorist to identify high-quality immunology research?

Another way to see the problem with taste is to remember that, by assumption, most scientists have pretty good taste. Therefore, within any given field, good-taste work is disproportionately likely to have been explored. That doesn’t mean that everything in good taste has been done, just that low-hanging fruit is likely to have been picked. It follows that if you’re looking for something new, it’s worthwhile to spend a lot of time exploring bad-taste ideas. Most other things being equal, they’re more likely to be the important ones.

For straightforward work, all the above considerations lean in favor of judging via taste. It’s only in the search for unusual (can I type “paradigm changing” without vomiting?), but high-quality, work that using taste becomes a handicap.