Derivation of Einstein-Hilbert Lagrangian

Question

How did Hilbert derive the Lagrangian $\frac{1}{2 \kappa} R \sqrt{-g}$ for the Einstein-Hilbert action?

If it was defined rather than being derived to be that way, then what was the behind it?

How did Hilbert conclude it was the right Lagrangian before deriving the Einstein equation?

Answer 1

General Relativity (GR) has already been proved to give very good predictions of the large scale physics in our Universe. However, GR is not a specific action, such as the Einstein-Hilbert action, it is something much profound.

GR is the union of space and time into the space-time, which forms a manifold that has a metric associated to it, and presents diffeomorphism invariance. In order of calculate any quantity throughout that manifold, we will always need to make a map of the quantity to a flat manifold (Minkowski space-time), and then undo this map. This is achieved through including the measure $\sqrt{-g}$ in the action. Therefore, not everything in the gravitational sector is randomly chosen.

Once all GR is derived, we need to state how does the curvature of the space-time react to the matter that we put in it. This is something that we need to guess, in the same way that we are guessing the Standard Model. Einstein and Hilbert chose that the gravitational sector would be given by the Ricci Scalar, because it would lead to a direct dependence between curvature and energy in the universe. However, this was just chosen for simplicity, and it has shown to work to an accuracy that should have already given at least 4 more Nobel prices to Einstein for his derivation.

But there is more to this! Physicists have asked the same question you are asking many times, and it is a very wide field of study. Why the gravitational sector is like that? We don't really know, but there are many alternative versions. Here goes the some of them, in case you are interested:

• Scalar-Tensor Theories: These are the oldest alternatives, first described by Brans and Dicke as an explanation for the interesting ratio that it was discovered between the mass of the proton and the Planck Mass. Basically, these theories couple a scalar field to the Ricci Scalar, and can be understood as theories of gravity in which the Planck Mass changes with time. Read more here.
• F(R) Theories: These are quite straightforward, instead of just having the Ricci scalar in the gravitational sector, lets also include other couplings between the Ricci tensor. In this sense, the Einstein-Hilbert action corresponds to $F(R)=R$. These can be related to scalar-tensor theories through the use of the equations of motion of the system, but we still don't know if that approximation is valid including quantum effects.
• Horndeski Theories: These are the most generic gravitational sector that you can dream of. It can actually be made a little bit more generic through Beyond Horndeski theories or DHOSTs, but they are too complicated to actually give any output. F(R) theories and scalar-tensor theories would be a subsets of this models. You can see more about this here.

As you can see, there are a lot of alternatives being studied. The downsides is that most of them introduce new forces into the universe, which we are currently trying to test, but they can be hiding from us through screening mechanisms. All these theories are inside what it's called Modified gravity. This means that the Einstein-Hilbert action gets modified, but General Relativity is still present!