This post is long, so I recommend you look at the summary at the end first. I'll answer my original question by starting with a more general approach, using the info I wrote here:
Assume all motions in question are rotations about the origin. What's going on is that as long as your rigid body consists of points only along one line through the origin, your rotation matrix $T\in\textrm{SO}(3)$ will be underdetermined.
To uniquely determine $T\in\textrm{SO}(3)$, we need to know the action of $T$ on two linearly independent points $\vec{x}_{1}, \vec{x}_{2}$. If we know $T\vec{x}_{1}$ and $T\vec{x}_{2}$, then we can fully determine $T$. (The proof of this is left as an exercise to the reader.)
The problem in my example is, we do not have two linearly independent point particles. We only have one particle.
Suppose we are given a trajectory $\vec{r}(t)$ with constant radial component, meaning its magnitude $|\vec{r}(t)|$ is constant so that its motion can be described by rotations about the origin. Moreover, suppose $\vec{r}(0) = (1, 0, 0)$. What is the set of all possible angular velocities? We can answer this in several different ways.
1. Looking at the Rotation Matrices Directly
Given two differentiable maps $T, S:\mathbb{R}\rightarrow\textrm{SO}(3)$ such that $T(t)\vec{r}(0) = \vec{r}(t)$ and $S(t)\vec{r}(0) = \vec{r}(t)$, define $X(t) = S(t)^{-1}T(t)$. This is a special orthogonal matrix that fixes $\vec{r}(0) = (1, 0, 0)$. The only possibility here is that $X(t)$ is a rotation about the $x$-axis:
$$ X(t) = \begin{pmatrix}
1 & 0 & 0 \\ 0 & \cos\phi(t) & -\sin\phi(t) \\ 0 & \sin\phi(t) & \cos\phi(t)
\end{pmatrix}. $$
The two rotations $T(t), S(t)$ are related by $T(t) = S(t)X(t)$. As I explained in my other post, to get the angular velocity, we take $T'(t)T(t)^{-1}$. Plugging $T(t) = S(t)X(t)$ and rearranging things gives
$$ T'T^{-1} = S'S^{-1} + S(X'X^{-1})S^{-1} $$
so
$$ \Omega_{T}(t) = \Omega_{S}(t) + \operatorname{Ad}_{S}(\Omega_{X}(t)) $$
where the $\Omega$'s are the $3\times 3$ matrix versions of the angular velocities and $S(X'X^{-1})S^{-1} = \operatorname{Ad}_{S}(\Omega_{X}(t))$ is the adjoint representation action on
$$ \Omega_{X}(t) = \begin{pmatrix}
0 & 0 & 0 \\ 0 & 0 & -\dot{\phi}(t) \\ 0 & \dot{\phi}(t) & 0
\end{pmatrix}. $$
When we convert these to angular velocity vectors, we get
$$ \vec{\omega}_{T}(t) = \vec{\omega}_{S}(t) + S(t)\vec{\omega}_{X}(t) $$
where
$$ \vec{\omega}_{X}(t) = \begin{pmatrix} \dot{\phi}(t) \\ 0 \\ 0 \end{pmatrix}. $$
Given that $\vec{\omega}_{X}(t) \parallel \vec{r}(0)$, it follows that $S(t)\vec{\omega}(t) \parallel \vec{r}(t)$. So we finally got our result: if transformations $T(t), S(t)$ yield the same path $\vec{r} = \vec{r}(t)$, then the corresponding angular velocities differ by $\vec{\omega}_{T}(t)-\vec{\omega}_{S}(t) \parallel \vec{r}(t)$, i.e. the difference between $T(t)$ and $S(t)$ is a rotation about the axis parallel to $\vec{r}(t)$. If all our particles are aligned along a line through the origin, there is an additional degree of freedom of rotating about that line. This is exactly what our common sense would have told us. If there were an additional particle linearly independent from the first (equivalently, non-colinear with the first one and the origin) being rotated about the origin, then we could discriminate between $T(t)$ and $S(t)$ and only one of the matrices would have described the correct rotation.
2. Looking at Angular Velocity vs Velocity
A way simpler way to find the possible angular velocities is to consider that for a point particle whose distance from the origin is not changing, $\vec{\omega}(t)\times\vec{r}(t) = \dot{\vec{r}}(t)$. This isn't true in general, but we're assuming $|\vec{r}(t)|$ is constant. At any time $t$, we can find a positively oriented orthonormal basis $\vec{e}_{1}, \vec{e}_{2}, \vec{e}_{3}$ where $\vec{e}_{1}\parallel\vec{r}(t)$, and we find
$$ (\vec{\omega}(t) + a\vec{e}_{1} + b\vec{e}_{2} + c\vec{e}_{3})\times\vec{r}(t) = \dot{\vec{r}}(t) - br(t)\vec{e}_{3} + cr(t)\vec{r}_{2}. $$
This equals $\dot{\vec{r}}(t)$ only when $b, c = 0$. Hence the only degree of freedom we have for the angular velocity is in $a\vec{e}_{1} \parallel \vec{r}(t)$.
3. Looking at the Moment of Inertia
Lastly, I want to think about this underdetermination of $\vec{\omega}(t)$ from the point of view of angular momentum and the moment of inertia. Surprisingly, the fact that $\vec{\omega}$ is undetermined in some cases is not a problem for calculating the moment of inertia at all. Angular momentum is always well-defined as $\vec{L} = \vec{r}\times\vec{p}$, yet $\vec{\omega}$ is not in our case. The implication of this is that the moment of inertia tensor $\overleftrightarrow{I}$ is degenerate. Given two distinct $\vec{\omega}_{1}, \vec{\omega}_{2}$ describing the same motion of the particle, we find $0 = \overleftrightarrow{I}(\vec{\omega}_{1}-\vec{\omega}_{2})$. One of the eigenvalues of $\overleftrightarrow{I}$ must be zero in that case. For this reason it's worth examining the moment of inertia tensor.
In this part, take the mass of the particle as $m$. Since $\overleftrightarrow{I}$ is a symmetric matrix, it follows by the spectral theorem for real symmetric matrices that there is an orthonormal basis in which $\overleftrightarrow{I}$ is diagonalized. Thus, we will ignore the off-diagonal entries and look at the moment of inertia tensor only in its diagonalized form:
$$ \overleftrightarrow{I} = \operatorname{diag}(I_{xx}, I_{yy}, I_{zz}) $$
where
$$ I_{xx} = \sum m_{i}(y_{i}^{2}+z_{i}^{2}),\quad I_{yy} = \sum m_{i}(x_{i}^{2}+z_{i}^{2}),\quad I_{zz} = \sum m_{i}(x_{i}^{2}+y_{i}^{2}). $$
If we had a particle located exactly at the origin, then all three $I_{ii}$'s would be zero. Ok, so that covers one case where $\overleftrightarrow{I}$ is degenerate.
If there is any mass outside the origin (either as a point mass or as a region of mass), then there is mass around a point $(x, y, z)$ where at least one of $x, y, z$ is nonzero. But then at least two of $y^{2}+z^{2}$, $x^{2}+z^{2}$, $x^{2}+y^{2}$ are nonzero, so at least two of $I_{ii}$'s are nonzero. The only possibilities for the moment of inertia matrix are that its null space is of dimension $3$ (in which $\overleftrightarrow{I}=0$), $2$, or $0$. The first case is trivial and the third case means $\vec{\omega}$ is uniquely determined.
In the second case, exactly one of the eigenvalues is zero. Let's say it is $I_{xx}$. In that case, all of our masses $m_{i}$ must have $z_{i}^{2}+y_{i}^{2}=0$. Hence, this scenario involves all of our mass lined up along the $x$-axis. Moreover, the null space of our $\overleftrightarrow{I}$ will be the span of $\vec{e}_{x}$. Consequently, if any two angular velocities $\vec{\omega}_{1}, \vec{\omega}_{2}$ differ by an $x$-component (along the line that the mass is lined up), then $\overleftrightarrow{I}\vec{\omega}_{1} = \overleftrightarrow{I}\vec{\omega}_{2}$. Now for general Cartesian coordinates, the same applies but the direction is along any line through the origin rather than the $x$-axis.
This is precisely what we've said initially: $\vec{\omega}$ is underdetermined if and only if our mass is distributed along a 1D line through the origin. The angular velocity has an extra degree of freedom along that axis and the moment of inertia tensor has an eigenvector along that direction with eigenvalue zero. Note that the angular momentum continues to be well-defined however.
Which Choice of Angular Velocity is Best?
Given that in the case of a point mass or a mass distribution along a line through the origin, the angular velocity can be underdetermined, we might ask, which choice of angular velocity vector is the best? The freedom in choosing an angular velocity vector is equivalent to the freedom in choosing the axis of rotation in our case, so we can equally well ask, which choice of axis of rotation is the best?
The answer is that it really depends on the circumstance. Moreover, this underdetermination is not a big deal. In the case of 3D extended rigid bodies, we don't have this underdetermination, and in the case of 1D mass along an line through the origin, we can just pick something and move on.
In the case of a single particle, we may set
\begin{align}\tag{$1$}
\vec{\omega} = \frac{\vec{r}\times\dot{\vec{r}}}{r^{2}}.
\end{align}
This is the unique angular velocity vector for which $\vec{\omega}\cdot\vec{r} = 0$, that is, there is no component of it along the radial direction (which was the source of our underdetermination). This formula also parallels the formula for orbital angular momentum $\vec{L} = \vec{r}\times\vec{p}$. There are a couple of precautions we have to keep in mind, however.
First, this formula only works for a point particle. It should not be used for rigid bodies with two or more linearly independent points. For example, we can imagine a cylinder such that
- its axis of symmetry, or longitudinal axis, lies within the $xy$-plane,
- the longitudinal axis is going through the origin, and
- the longitudinal axis is rotating counterclockwise from the top view down on the $xy$-plane.
(In other words, imagine the cylinder is circularly orbiting the origin and it always points to the origin.) In that case it has $\vec{\omega} = c\vec{e}_{z}$. Now suppose the cylinder is rotating about its longitudinal axis in addition to its orbit around the origin as described initially. Then a couple of things happen:
- The angular velocity is no longer in the $z$-direction. Instead it is of the form $\vec{\omega} = a\vec{e}_{r} + c\vec{e}_{z}$ where $\vec{e}_{r}$ is the unit vector pointing to the cylinder along the cylinder's axis of symmetry. This new component $a\vec{e}_{r}$ was meaningless when we talked about a point particle in orbit, but it is not meaningless for 3D extended rigid bodies.
- If we calculate $\vec{\omega}$ for each point on the cylinder according to $(1)$, then we will obtain different $\vec{\omega}$'s for different points on the cylinder! This means the formula $(1)$ can't work for 3D extended rigid bodies even in principle. Instead, you need to define angular velocity as outlined in my post here, starting by looking at the rotation matrix (describing angular displacement over time), differentiating it, and extracting the angular velocity vector according the recipe in the post.
Let me also say, even in the case where you're dealing with a single particle in orbit around the origin, formula $(1)$ doesn't really give you the most convenient axis of rotation. In my original example where the particle took trajectory $\vec{r}(t) = (a\cos ct, a\sin ct, 0)$, formula $(1)$ gave us a constant angular velocity vector $\vec{\omega}(t) = c\vec{e}_{z}$. Any other choice would have resulted in a changing $\vec{\omega}$ with a precessing axis of rotation. This convenience given by $(1)$ isn't always the case.
Example 1. Let $\vec{r}(t) = (\cos ct, \sin ct, 1)$. Then formula $(1)$ gives
$$ \vec{\omega}_{1}(t) = (-\frac{c}{2}\cos ct, -\frac{c}{2}\sin ct, \frac{c}{2}), $$
and this is non-constant and it clearly has a precessing axis of rotation. On the other hand, we can exploit the extra degree of freedom in choosing our angular velocity vector to obtain
$$ \vec{\omega}_{2}(t) = (0, 0, c), $$
which is constant and much more intuitive in describing what is going on (the particle is orbiting around the $z$-axis. This is an example where the formula $(1)$ is disadvantageous even for the case of a point particle.
Summary
- For structureless point particles, or mass distributed along a 1D line through the origin, the angular velocity vector describing the orbit of the mass around the origin will be underdetermined.
- This turns out not to be a problem when you're using the moment of inertia tensor to convert from angular velocity to angular momentum.
- The ambiguity goes away once you have any mass placed on at least two linearly independent points such as in the case of 3D extended rigid bodies.
- When you're dealing with an orbiting particle, sometimes you can exploit the ambiguity to obtain a convenient angular velocity vector. Sometimes you can't. In any case, you just have to be consistent in how you choose your axis of rotation in that scenario.
