The answer is the forces. The nuclear force, the residual strong force, is holding the neutrons and protons together in the nucleus. Now, the strong force is short ranged. Actually, on the very short scale, it becomes repulsive (this keeps neutrons apart), and after a certain distance it is attractive (this keeps them still close in a nucleus).
Now, if you start having more and more neutrons, they will tend to decay into protons, because they are rather unstable. So your neutron rich nucleus will have to have protons too. This is a little bit more complicated, because the isospin of the neutron and the proton complement, and add an extra force to hold the nucleus together.
So there are three forces in the nucleus holding it together:
residual strong force holding the neutrons and protons together
isospin coupling makes nucleus with neutrons and protons more stable (then just neutrons)
EM force keeps protons apart
Now the stability of your nucleus will depend on these forces, as long as they equalize out, it will be stable.
The problem is the range of these forces. The EM force is repulsive between the protons, and keeps them apart, but the strong force is relatively short ranged.
When you add more and more neutrons, you will grow in size. It is because the strong force is repulsive on very short scales. This keeps the neutrons and protons apart too. And the EM force keeps the protons apart too.
So when you grow in size, the nucleus gets bigger, and the residual strong force will after a while not be able to equalize out the EM repulsion. After you add more then 118 to 150 neutrons (and the protons needed because of isospin), the nucleus will fall apart.