Imagine there is me, Earth and some other guy. Me and a guy move parallel to each other at the speed of 1000m/s relative to Earth.
I am so fit that my mass is 0.5kg, so when a force of 1N in the direction of my movement acts on me for 1s, I accelerate to 1002m/s and in that time I travel 1001m (all relative to Earth).
The work is force multiplied by displacement, so 1N*1001m=1001J
But from the guys perspective I was still, and after the force I moved just 1m relative to him.
And here the work is just 1N*1m=1J.
So is it that the work depends on the reference point, or the work is the difference between displacement and displacement if there was no force?