I have Yaw, pitch, and roll angles in that order (Euler 321) to apply to a body reference frame in cartesian coordinate system. I want to know what the body reference frame vector coordinates are after those angles are applied.
I start with a orthonormal reference frame $R$, apply the rotation matrices $R_Z$ then $R_Y$ and $R_X$, those being: $$R_{X}(\theta) = \begin{pmatrix}1 & 0 & 0 \\ 0 & \cos \theta & -\sin \theta \\ 0 & \sin \theta & \cos \theta\end{pmatrix},\qquad R_{Y}(\theta) = \begin{pmatrix}\cos \theta & 0 & \sin \theta \\ 0 & 1 & 0 \\ -\sin \theta & 0 & \cos \theta\end{pmatrix},\qquad R_{Z}(\theta) = \begin{pmatrix}\cos \theta & -\sin \theta & 0 \\ \sin \theta & \cos \theta & 0 \\ 0 & 0 & 1\end{pmatrix}.$$ I have assumed rotation matrices rotate about the corresponding axis of the reference frame to which it's applied, not of the coordinate system, however Matlab seems to say otherwise when I plot the reference frames before and after: yaw is fine because it is the first operation, but during pitch Ybody is not invariant for example.
In contrast, I tried to replace each of the rotation matrices previously defined in: $$RotatedRefFrame=R_XR_YR_ZRefFrame$$ By the rotation matrix about an arbitrary axis: $$R_u = \begin{bmatrix} \cos \theta +u_x^2 \left(1-\cos \theta\right) & u_x u_y \left(1-\cos \theta\right) - u_z \sin \theta & u_x u_z \left(1-\cos \theta\right) + u_y \sin \theta \\ u_y u_x \left(1-\cos \theta\right) + u_z \sin \theta & \cos \theta + u_y^2\left(1-\cos \theta\right) & u_y u_z \left(1-\cos \theta\right) - u_x \sin \theta \\ u_z u_x \left(1-\cos \theta\right) - u_y \sin \theta & u_z u_y \left(1-\cos \theta\right) + u_x \sin \theta & \cos \theta + u_z^2\left(1-\cos \theta\right) \end{bmatrix}$$ Taking the axes off the reference frame, and it works.
My question is therefore: do rotation matrices rotate about BODY axes of the geometry rotated, or INERTIAL axes? The former case would mean I have an error in my code, but simplifying it to the edge of being just the mathematics presented and thoroughly debugging it step by step hasn't revealed any error.