If we take a long stick which looks perfectly straight in curved spacetime on the surface of the Earth. Then if we take this stick into space away from the massive celestial bodies, it should become curved?
It certainly can happen that way, but it does not need to happen that way. It all depends on the details of the experiment which you did not specify.
Suppose that we are measuring "perfectly straight" as follows: we have a small tunnel through the axis of the stick with non-reflective walls. We will shine a laser through the tunnel aligned with the entrance of the tunnel, and if it passes through then the stick is straight, if the laser is absorbed we can detect where it is absorbed and know that it is curved in the opposite direction.
Now, suppose that we start with the stick hanging vertically. If it is straight on earth then it will also be straight in space.
Suppose that we start with the stick horizontal supported only on one end (a cantilever beam). In that case a stick that is straight in space will be curved downwards on Earth, and the light will hit the top. Conversely, a stick that is straight on Earth will be curved in space, and the light will hit the "bottom". There is material stress in this stick so the amount of bending depends on the stiffness of the material.
Suppose that we start with the stick horizontal supported on each end. In that case a stick that is straight in space will be curved upwards on Earth, and the light will hit the bottom. Conversely, a stick that is straight on Earth will be curved in space, and the light will hit the "top". There is material stress in this stick so the amount of bending depends on the stiffness of the material.
Suppose that we start with the stick horizontal and supported uniformly across the length of the stick so that there is no material stress in the stick. In that case a stick that is straight in space will be curved upwards on Earth, and a stick that is straight on Earth will be curved "downwards" in space. Since there is no material stress the amount of bending does not depend on the stiffness of the material.
Finally, suppose that we are orienting the stick horizontally and dropping it so that it is in free fall near the surface of the Earth. Further, suppose that the stick is long enough so that tidal effects (spacetime curvature) are non-negligible. In that case, a stick that is straight in space will be curved near the earth and the light will hit the bottom. There is stress in the rod falling near the Earth due to tidal effects so the bending depends on the stiffness of the material.