A lot of twin paradox confusions can be clarified by realizing that the elapsed time for an object is just the (suitably defined) length of its spacetime path. That is, think about the full 4-dimensional spacetime. In the classic formulation, one twin's path is a straight line in the time direction while the other is not straight and deviates in one of the spatial directions. (You can find illustrations on the Wikipedia page, for example.)
In this perspective, the twin paradox is really just a question of the following. If we pick two spacetime points and draw two different curves between them, which curve is longer and which is shorter? It should be clear that there is no ambiguity here: every curve has a well defined length.
Without acceleration, you are limited to drawing straight lines (technically geodesics). In flat spacetime, there is only one straight line between two points, but in curved spacetime there could be many. In general, if two unaccelerated twins experience different amounts of elapsed time, it is because they crossed different regions of curved spacetime. So the spacetime curvature is what makes the situation asymmetrical.
Note that symmetrical formulations of the unaccelerated twin paradox are also possible. In that case both twins will experience the same amount of time. For example, imagine two satellites on circular orbits at the same radius (and in the same plane) but orbiting in opposite directions, so that they can compare clocks every half orbit. The paths of these two satellites are symmetrical, and it is also clear that the same amount of time elapses for both.