Although the "Lovelock gravity" reference in the comments probably takes care of this, it is worth pointing out here explicitly that the three quite natural requirements on the action functional $$ S[g]=\int_M L[g] $$of gravity, namely
- the Lagrangian must be equivariant with respect to diffeomorphisms;
- the metric is the only field that appears in the Lagrangian;
- the field equations are second order in the metric;
are quite restrictive. Lovelock's theorem gives the most general such Lagrangian, and it is a certain combination of curvature invariants, but in a four dimenional space, the only possiblity is $$ L[g]=(aR+b)\mu,\quad\mu=\sqrt{\mathfrak g}\mathrm d^4x. $$
For example the so-called Gauss-Bonnet (GB) term is also allowed, but in four dimensions its integral is a topological invariant, so it does not influence the Euler-Lagrange equations at all. In a spacetime whose dimension is larger than four, the GB term gives a genuine generalization of the Einstein-Hilbert action.
To answer one of OP's question directly, namely
Why not a generic function $f(R)$?
Such a generic function will give a fourth order field equation for gravity. Also linked in the comments are references to $f(R)$ gravity, which is a modification of GR that is actively researched, so people do consider such theories.
We generally however want to avoid higher order theories for two reasons. One is that Newtonian gravity is second order, and the other is that higher order theories tend to suffer from certain instabilities called Ostrogradski ghosts.
Neither problem is completely exclusionary, for example higher derivative parts tend to be suppressed at low energies, so they can still give acceptable Newtonian limits and Ostrogradski instabilities may be avoided if the Lagrangian is sufficiently degenerate, which is pretty much always the case for gravitation theories (the degeneracy, not the ghost avoidance). One must then do a case-by-case investigation via eg. the Dirac-Bergman formalism to determine whether the theory is still unstable.