The radius of a typical nucleus, in terms of number of nucleons, is $R=A^{1/3}R_0$ where $A$ is the mass number and $R_0$ is $1.25\space\rm{fm}$, with typical deviations of up to $0.2\space\rm{fm}$ from this value.
At least part of it seems to be that nuclei aren't always spherical (if you can even call anything at that scale spherical), but are there other factors, like electrostatic repulsion between protons and the fact that multiple nucleons can fit in the same shell?
See the answer in the following thread, which asks a similar question: The central density of a nucleus remains roughly constant?
The fact that the volume should scale linearly with the number of nucleon means that the nucleons can be thought of as somehow incompressibly "packed," which is only a reasonable assumption in the first place because the strong force is short range (i.e., nucleons are really only significantly interacting with nearest neighbors, which is how, e.g., marbles in a bowl would interact). This is an accurate (but not perfect) simplification, which I think should be at least part of the reason for this.
To enlarge slightly upon Nicholas Rui's answer (+1), if one assumes that nucleons are as closely packed (hexagonally close) as possible, then it is geometrically impossible for nuclear energy shells to be spherical because the HCP network is itself not spherical. Therefore it will be geometrically impossible for every nuclear configuration to settle into a binding energy minimum, which means that there will be discontinuous jumps in nuclear binding energy as the number of nucleons is increased.
