Most examples I've seen describe the (gravitational) potential energy e.g. with an example of a ball thrown upwards, and explain how the sum of the ball's kinetic and potential energy is constant at all times. This is clear to me.
Based on the formula $U=mgh$ my understanding is that an object resting on the ground has a potential energy of 0 (as $h=0$).
What I can't understand is the following setting. Imagine there are three platforms of various heights (see the attached picture). I'm standing on the middle one holding a ball. As the ball is above the "ground", it has some potential energy that is dependent on the height $h_0$. If I would move the ball to the right, so that it is completely over the rightmost platform, its potential energy would be different now, as the distance to the ground has changed (it is now $h_1$). But now, if I would move the ball the same distance but in the other direction, its potential energy would be yet different, as it would be proportional to the $h_2$. Importantly, in both cases I would need to exercise exactly the same amount of work to move the ball. This, of course, is absurd, as the total energy of the system should be constant, and in this case I could end up with different energies based on whether I move the ball left or right. So, what am I missing?
While writing this question I realized that one possible solution to this problem is to measure object's height relative not to the "ground", but to the gravitational center of the system. Is this how the potential energy is actually defined, and the "height above the ground" is just a simplification to explain the concept?