Contrary to your claim, chirality is in fact a purely mathematical conception, while handedness has also physical interpretation.
In elementary particle physics, chirality just defines the transformation law of the given field under the Lorentz group. For example, Dirac spinor is composed from two 2-component spinors transforming in different ways under the Lorentz transformations, while the chirality projectors $\frac{1\pm \gamma_{5}}{2}$ projects the Dirac spinor on a space of one of these two spinors. As long as the particle which is described by the Dirac spinor is massive, chirality isn't conserved in time even in free theory. However, it is Lorentz invariant.
Handedness, or (more correctly) helicity, is the projection of total angular momentum on the direction of motion. For massive particle with spin $s$, there are $2s+1$ possible values of helicity. Massless particles can only be characterized by helicity (not by spin), which can be $h$ or $-h$. The state with helicity $-h$ is called the left-handed state, while the state with helicity $+h$ is called the right-handed state. For free theory, helicity is conserved in time, but is not Lorentz invariant, which obviously follows from the definition.
As you can see, in general helicity is completely unrelated to chirality. But in massless case the value of helicity, apart from its physical sense, also defines the Lorentz transformation law for the given field, and therefore it coincides with chirality. By the other words, the eigenvalue of helicity can be directly compared with the eigenvalue of chirality in massless limit.