An $SU(2)$ symmetry, a priori, has absolutely nothing to do with spatial rotations. For instance, the symmetry between protons and neutrons is described by isospin, an $SU(2)$ symmetry that treats the proton and neutron just like the spin up and spin down states of a spin $1/2$ particle. Similarly there is weak isospin, also described by $SU(2)$, which relates the electron with the electron neutrino. Neither of these are related to rotations, except for the math being the same; you cannot do a physical rotation to turn a proton into a neutron. (However, the analogy with spin is so useful for conceptualizing what's going on that they all have the word 'spin' in their names.)
If a textbook says "we've identified an $SU(2)$ symmetry, so it must physically correspond to rotational symmetry", then that book is being sloppy. The argument should be phrased in reverse. Below I'll show how that goes very explicitly.
We start with experimental data that we want to understand. For example, suppose we want to model the precession of the magnetic dipole moment of a nucleus in a time-varying magnetic field. (It turns out this dipole moment is proportional to the spin, so this is exactly what you're asking about.) To make a quantum mechanical model we must define a Hilbert space and a Hamiltonian, as well as operators $\mu_i$ corresponding to the components of the magnetic dipole moment.
Of course, there is no unique mathematical prescription for this. For example, you may choose the Hilbert space to be zero-dimensional, but then it clearly wouldn't fit the data. It turns out that for some nuclei the model works if we choose the Hilbert space to be two-dimensional, with the states $|\uparrow\rangle$ and $|\downarrow\rangle$ corresponding to the dipole moment pointing vertically up and down. In other words, this defines $\mu_z$ as
$$\mu_z |\uparrow \rangle = \mu_0 |\uparrow\rangle, \quad \mu_z |\downarrow\rangle = - \mu_0 |\downarrow \rangle.$$
For convenience we'll ignore spatial degrees of freedom. You can think of the nucleus as nailed down at the origin, if you want.
Next, we define rotation operators that physically rotate the system around. We know that classically, the rotations of space form a group $SO(3)$. Because of issues with quantum phases, this means the rotation operators must necessarily be a representation of the group $SU(2)$. This is a bit weird, but something you get used to after a while -- there is generally a turn-the-crank mathematical procedure for figuring out which group to use in the quantum case.
However, we still don't know which representation of $SU(2)$ it is. For example, the rotational operators could conceivably all do nothing -- that is exactly the right choice for isospin symmetry (ignoring the spins of the proton and neutron) because you can't rotate a proton into a neutron. But it's not the right choice for the magnetic moments, because we can observe that tilting the magnet and running the experiment again gives a different result.
We also know that a rotation about the $z$ axis should fix $|\uparrow \rangle$ and $|\downarrow \rangle$, again by observation, while $180^\circ$ rotations about the $x$ and $y$ axes should interchange these states. (Everything here is just up to phases.) We can't proceed further and get explicit expressions because they will vary depending on the phases of the states. But in the standard phase conventions, one can continue with this logic to show that the $\mu_i$ operators must all be proportional to $\sigma_i$, and the rotation operators are exponentials of $\sigma_i$. This is described in detail here.
This is a pretty long and very explicit argument to make one simple point: in physics, we do not blindly do mathematics and put the physical interpretation in at the end. We start with a physical system we want to describe and define mathematical objects accordingly. We know that rotations must be described by $SU(2)$ by general principles, so we define a Hilbert space with a representation of $SU(2)$. Which one? Whatever works.
