According to the theory of quantum mechanics, if a spin state is prepared along axis "x", and then measured along axis "z", then the result of the spin projection is probabilistic: half of the times it will point up, and the other half down.
My question is the following: in doing this experimentally, how do we know that this randomness is not caused by the state preparation itself, or by a shot-to-shot fluctuation in the coupling to the measurement device?
How do we know that the randomness is not caused by the state preparation itself?
It depends what you mean here. In some sense it is caused by the state preparation: the "state preparation" generates the quantum state, which is the cause of the randomness in the measurement outcomes.
But what you probably mean is: how do we know that the randomness in the measurement outcomes is not just due to some "classical uncertainty", that is, how do we know that the state preparation procedure is not just generating half the time the state $\lvert\uparrow\rangle$ and half the time the state $\lvert\downarrow\rangle$?
In other words, how can we make sure that the received state is a pure state $\lvert\psi\rangle=\frac{1}{\sqrt2}(\lvert\uparrow\rangle+\lvert\downarrow\rangle)$ and not a mixture of the form $\rho=\frac{1}{2}(\lvert\uparrow\rangle\langle\uparrow\rvert + \lvert\downarrow\rangle\langle\downarrow\rvert)$? The answer is that, by just looking at the measurement outcome in a fixed basis, you can't. But what you can do is to see what you measure after a rotation of the state.
In this simple case you can for example apply an Hadamard gate to the state (what this means depends on what kind of system is being considered), and measure after this gate. An Hadamard gate is the unitary transformation $H=\frac{1}{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$, and you can easily check that $H\lvert\psi\rangle=\lvert\uparrow\rangle$. On the oher hand, as you can verify, $H\rho H = \rho$. What this means is that after the Hadamard, the measurement statistics will tell you whether your initial state was $\lvert\psi\rangle$ or $\rho$: if you always get the result corresponding to $\lvert\uparrow\rangle$ then you had the pure state $\lvert\psi\rangle$ (that is, a spin state prepared along the X axis), while if you still get half the time a result and half the time another result, it means that your initial state wasn't really what you expected (that is, the state preparation procedure was "cheating" by just feeding you sometimes $\lvert\uparrow\rangle$ and sometimes $\lvert\downarrow\rangle$).
While in this case it's quite simple to distinguish the two cases, the more general problem of discriminating one state from another is nontrivial, and a vast literature has been devoted to its study (just google for quantum state discrimination to get some references).
If one is instead only interested in certifying the purity of a state, like in your case, then the problem is somewhat easier than the general quantum state discimination problem, but still not really trivial in general. One interesting thing is that it turns out that the statistics generated by a random pure state is different than that generated by a random mixed state. This means that one can certify the purity of a state by just applying a random evolution to the state and looking at the output distribution of probabilities (see Beenakker et al. 2009 and Enk and Beenakker 2011)
or by a shot-to-shot fluctuation in the coupling to the measurement device?
This doesn't change much the above argument. You can just model this by saying that the measurement device (that is, the measurement basis) is fixed, while the state that is being prepared changes from shot to shot. Or you can do the opposite and say that the state prepared is fixed while the measurement basis (your "shot-to-shot alignment") changes.
A case in which a similar situation can make sense is the following: I have the state $\lvert\psi\rangle$, and this state undergoes a random unitary rotation (modeling your "shot-to-shot fluctuation") before the measurement. What can you expect to learn about $\lvert\psi\rangle$ in this situation? The answer is: not much. In particular, assuming the rotation is truly random, you can never say whether your state is $\lvert\uparrow\rangle$ or $\lvert\downarrow\rangle$ or $\frac{1}{\sqrt2}(\lvert\uparrow\rangle+\lvert\downarrow\rangle)$ or whatever else. What you can still say is whether the state is pure or not, thanks to the same protocol employing random rotations mentioned above.
There is a beautiful theorem of Kocher - Specker showing that principles of quantum mechanics are incompatible with explanation you thought about. Take it here: https://plus.maths.org/content/john-conway-discovering-free-will-part-ii
We absolutely can't rule this out. Indeed, in the de Broglie-Bohm pilot wave interpretation of quantum mechanics, this is exactly what happens: there really is a deterministic 'actual' state, but there's also a hidden pilot wave that we can never experimentally control or observe, which gives rise to the apparent randomness. This doesn't contradict the Bell inequality because the pilot wave is nonlocal. And of course, pilot wave is exactly equivalent to every other interpretation of quantum mechanics, so every experiment comes out the same.
However, most physicists don't take pilot wave seriously. The sole philosophical appeal of pilot wave is that you can get a deterministic state, but the cost is much worse: you pick up a nonlocal, exceedingly complicated hidden variable that cannot be measured or controlled even in principle. Why keep that thing around? Just accepting that there is chance seems much simpler, and certainly is much more computationally efficient.
