By assuming that energy is expended in space travel you are implicitly assuming that the spaceship is accelerating. This is just Newton's law. This frame is no longer inertial. Thus you are comparing an accelerated frame to an inertial frame. However nothing in principle of relativity stops you from differentiating an accelerated frame from an inertial frame.
On the other hand if you talk about a uniformly moving space ship then it is perfectly alright to see either one of them as 'moving'.
$\underline{Note -}$
Incidently the answer to your question is exactly what resolves Twin paradox. Let the twins be T1,T2.
$\textbf{Setup}$ - T1 has started moving in a spaceship on Jan 1,2018 while T2 stays back at earth. They have decided that they will meet again in 50 years(taking into account time dilation and all that). So, when they meet who will be younger?
$\textit{We know moving guy should age slowly and hence stay younger}$
$\textbf{Paradox}$ - By relativity both frames are moving and hence when they meet both should see the other as younger. Hence a paradox! The setup is symmetrical or is it?!
$\textit{Who is actually younger?}$
If you think carefully you will immediately notice that just like your question T1 had to first accelerate to reach a uniform velocity. So the set-up is not symmetrical. Again to compare their ages they have to meet and reach same frame again. For this to happen the spaceshi guy has to turn around(again acceleration,non-inertial) and in the decelerate(again non-inertial).
Thus we can confidently say that T1 is in non-inertial frame while T2 is in inertial frame. But we can apply theory of Special relativity only to inertial frames. Hence we are FORCED to work in T2's frame and conclude that it is T1 who should be actually younger.
This dramatic experiment has been done using particles called $muons$ and experimental results fit beautifully with the theoretical predictions.
$\large \textit{Hail, O Einstein!!}$