All the answers you have read so far are models that represent reality as best as possible. The basic starting point is the empirical arrangement of the chemical elements in the periodic table. There we see a periodicity of, 2, 8, 8, 18 and 18 elements (and other higher periods), each ending with a noble gas. The elements are distinguished by the continuous increase of electrons by one electron each. And they are similar for certain numbers of electrons. Good examples are carbon and silicon, fluorine and chlorine or neon and argon.
Absolutely fundamental is the Pauli exclusion principle. It states that electrons in an atom differ in at least one quantum number.
Much less evidence-based are the shell regions, which are defined by the use of special Spherical harmonic. The development of these harmonics was as follows.
Periodicity of hydrogen spectral lines.
It is a fiddle to get a formula for the periodicity of the emission lines of hydrogen. Balmer was the first to succeed in 1885 and Rydberg made an extended formula out of it. All for hydrogen, mind you.
Bohrs shell model of atoms
The first Bohr orbit is filled when it has two electrons, which explains why helium is inert. The second orbit allows eight electrons, and when it is full the atom is neon, again inert. The third orbital contains eight again, except that in the more correct Sommerfeld treatment (reproduced in modern quantum mechanics) there are extra "d" electrons. ... The irregular filling pattern is an effect of interactions between electrons, which are not taken into account in either the Bohr or Sommerfeld models and which are difficult to calculate even in the modern treatment.
An important piece of the puzzle for Bohr's atomic model was the application of the Rydberg formula.
In fact, Bohr's derivation of the Rydberg constant, as well as the concomitant agreement of Bohr's formula with experimentally observed spectral lines of the Lyman, Balmer, and Paschen series, and successful theoretical prediction of other lines not yet observed, was one reason that his model was immediately accepted.
In reality, however, only hydrogen and hydrogen-like elements can be calculated with the Rydberg formula.
This formula can be directly applied only to hydrogen-like, also called hydrogenic atoms of chemical elements, i.e. atoms with only one electron being affected by an effective nuclear charge (which is easily estimated). Examples would include He+, Li2+, Be3+ etc., where no other electrons exist in the atom.
But the Rydberg formula also provides correct wavelengths for distant electrons, where the effective nuclear charge can be estimated as the same as that for hydrogen, since all but one of the nuclear charges have been screened by other electrons, and the core of the atom has an effective positive charge of +1.
Finally, with certain modifications (replacement of Z by Z − 1, and use of the integers 1 and 2 for the ns to give a numerical value of 3⁄4 for the difference of their inverse squares), the Rydberg formula provides correct values in the special case of K-alpha lines,...
I have quoted this at such length to show the limitation of the Rydberg formula.
Nevertheless, three and later 4 quantum numbers were formed from it.
Some spherical harmonics are shown in the following picture.

The harmonics on which our picture of the quantised atomic structure is based form lobes along the Carthean coordinate system.

However, there are a variety of other solutions. One of them is shown in the picture in the second row on the left (l=3, m=2, l-m=1).
In this special case, exactly 8 electrons can be calculated. You have to imagine it like a cube, on whose 8 corners the positions of the electrons are located. And it is absolutely amazing that the electron spins can point alternately inwards and outwards for each corner without two nearest electrons having the same spin (corresponding to the + and - signs in the picture above).
Why second shell can have eight electrons?
Because the electron properties together with the nucleus properties give stable situations in nature. The Periodic table of elements is the foundation for any model and the model with 4 quantum numbers is a very complicated model with a lot of exceptions and assumptions that later turned out to be untenable.