Epistemic note: I am only minimally knowledgable in matters related to cosmology. My impression is that the observation below is “well known to those who know it well”.

There’s a standard story of the zero-point energy of the standard model—that is, the energy density of the vacuum state of the standard model. We’re going to try to estimate it from theory, and then estimate it from cosmological observations, and the two results differ by \(120\) orders of magnitude.

The estimate from theory isn’t complicated. Let’s pretend that the standard model consists of a single scalar particle of mass \(m\). The energy is just the sum of energies in every available momentum mode. Recalling that the vacuum energy of a harmonic oscillator with gap \(\omega\) is \(\frac\omega 2\), we find: \[ \epsilon = \int d^3 k \frac{\sqrt{m^2 + k^2}}{2} \text. \] Alas, this integral diverges! What can we do? The standard approach here is to notice that eventually, the momenta \(k\) get large enough that Planck-scale physics (presumably quantum gravity) takes hold. We don’t know what happens there, but the integral along the way up is approximately \(\Lambda^4\), where \(\Lambda\) is the Planck cutoff scale. So we’ll just use that as a first approximation, neglecting whatever happens at the Planck scale.

This results in an unrealistically enormous zero-point energy: \(\sim 10^{100}\;\mathrm{g}\,\mathrm{m}^{-3}\). The estimate from cosmology is far smaller—a factor of \(10^{120}\). (An easy mnemonic is that it’s comparable to what you would get by using the cosmic microwave background to set the energy scale instead. This is, hopefully, a coincidence.)

At this point we’re in the land of many excuses. Here I want to make one narrow point about the calculation above: just from the standpoint of quantum field theory (that is, without needing to worry about cosmology, or quantum gravity, or any other difficult subjects), the calculation is already suspicious enough to probably be wrong.

To see why, let’s re-do the calculation, but now with a massless field. The energy density is \[ \epsilon = \int d^3 k \;|k| \sim \int_0^\Lambda dk \; k^2 |k| \sim \Lambda^4 \text. \] This is the same as it was before—no surprise, since \(m \ll \Lambda\). But now we have a far more immediate problem than some vague mumbo-jumbo about the cosmological constant. The free massless scalar field is a conformal field theory, meaning, among other things, that it has no intrinsic length scale. The true energy density of a free massless scalar field is \(0\). In that divergent integral above, we know what the correct regularization is. The integral must evaluate to zero.

Do not be deceived: I am implicitly invoking a somewhat questionable principle. It’s true that the energy density of a massless scalar field, at zero temperature, must be \(0\). But the real universe is not a CFT; the real universe has some sort of cutoff. How can we ignore that? Well, most properties of a continuum QFT are defined by a limiting process, whereby the cutoff scale \(\Lambda\) is lifted, and the masses (for example) of the QFT are defined by the limit \(\Lambda\rightarrow \infty\) of their values in the regularized theories. I am assuming that the same idea must also hold for the “true” energy density. In other words, the assumption is that when \(\Lambda\) is far larger than all other scales, the theory must look exactly like the theory with no cutoff at all. As far as I know, you’re free to disbelieve this!

Assuming you accept that principle, the energy density is \(0\) or (at finite \(\Lambda\)) very close. In fact every dimensionful quantity vanishes in a CFT, so we have \[ \epsilon = \frac{d\epsilon}{dm} = \frac{d^2\epsilon}{d^2m} = \frac{d^3\epsilon}{d^3m} = \frac{d^5\epsilon}{d^5m} = \cdots \text. \] The only one missing is the fourth derivative, which as a dimensionless quantity is permitted to be non-zero in a conformal field theory. As a result, we can integrate back the mass to approximate \[ \epsilon \sim -m^4 \] This is in retrospect unsurprising: after all, we had only one mass scale to begin with. (The sign is perhaps surprising, but I have nothing useful to say about that.)

The Planck energy is about \(10^{17}\) times the mass of the top quark, so this removes 68 of our 120 problematic orders of magnitude. In the context of the standard model and real-world cosmology, this resolves nothing. Any SM mass scale is still far too large to yield a sensible cosmological constant. It seems to me, though, that the oft-repeated story is considerably overstated.