I recommend avoiding "relativistic mass", it's not helpful. Worse: it seems to imply that there is an absolute rest frame so that moving near $c$ is somehow special because you have diverging mass. (You don't, you're moving near $c$ right now, in some frame...and nothing is different...that's "relativity").
I also recommend not thinking about the frame of a photon. PBS Spacetime, Fermilab, and so on yt videos are full of "but a photon doesn't experience time" questions. There is no inertial frame moving at $c$, and thinking there is only leads to confusion.
However: you can make progress with affine-parameter, $\xi$. Suppose a particle travels on a path $P(\tau)$ (parametrized by its proper-time, $\tau$). The 4-velocity is:
$$ u^{\mu} = \frac{dP}{d\tau} = \frac{dx^{\mu}}{d\tau}$$
from that, you can define 4-momentum:
$$p^{\mu} = mu^{\mu} = m\frac{dx^{\mu}}{d\tau} = \frac{dx^{\mu}}{d\xi}$$
where:
$$ \xi \equiv \tau/m $$
is a renormalized version of proper time (aka: the affine parameter).
Now for a massive particle moving slower than light:
$$ u^{\mu} = \frac 1 {\sqrt{1-\frac{v^2}{c^2}}}(c, \vec v) $$
and
$$ p^{\mu}=mu^{\mu} = \frac 1 {\sqrt{1-\frac{v^2}{c^2}}}(mc, m\vec v) =(E/c, \vec p)$$
is all good, but if $||v||=c$, the 4-velocity equation breaks, and you are stuck.
Now you apply the affine parameter ($\tau, m\rightarrow 0$, with $\tau/m$ fixed), because it allows you to get 4-momentum by skipping the 4-velocity step:
$$ p^{\mu} = \frac{x^{\mu}}{d\xi} = (||\vec p||, \vec p) $$
which becomes, in quantum mechanics:
$$ p^{\mu} = (\hbar\omega/c, \hbar\vec k) $$
with
$$ \omega = c||\vec k|| $$
So that's what you do at $v=c$. It's good to know, but it will not help in understanding any of the classic relativity paradoxes.
The notion that a photon has relativistic mass is deeply wrong. One related concern that pops up a lot is, "if a photon has no mass, how can it have momentum?". It is entirely based on the Newtonian concept of momentum.
If you look at a massive particle, the momentum per unit energy is:
$$ \frac p E = \frac{\gamma mv}{\gamma mc^2} = \frac v {c^2}\rightarrow \frac 1 c$$
where the arrow is as $\vec v \rightarrow c$.
Meanwhile, for a photon:
$$ \frac p E = \frac{\hbar k}{\hbar \omega} = \frac k{\omega} = \frac 1 c$$
The point is, mass just slows you down (literally), and makes it harder to get momentum out of energy. This is in complete contrast to our Newtonian intuition, which says:
$$p = \sqrt{2mE}$$
Other than that, thank you!
– ihateelectricalphysics Dec 04 '20 at 16:45