But why should matter curve space one way and not the other way?
The direction that spacetime is curved is determined by the sign of the energy density. If there were some exotic form of matter with a negative energy density then it would curve spacetime in the opposite direction than for ordinary matter with positive energy density.
At the same time, a manifold must locally be like Euclidean.
So what is spacetime really like according to general relativity?
In GR spacetime is modeled as a pseudo Riemannian manifold, with a metric that is determined by the matter content via the Einstein field equations.
The signature of the metric is (-+++), where - indicates measurements with clocks and + indicates measurements with rulers. At any location you would use one clock and three orthogonal rulers.
The path of an object in spacetime is called its worldline. An object with an accelerometer that reads 0 travels on a geodesic worldline, which is a “straight line”. This is the relativistic version of Newton’s first law.
Spacetime curvature represents tidal gravity. In the absence of gravity, if two nearby objects start off going at the same velocity and with 0 accelerometer readings, then the distance between them will not change. Geometrically this is the fact that in flat spacetime parallel straight lines always remain equidistant. In the presence of tidal gravity, such objects will not always remain equidistant. Geometrically this is straight lines that are parallel at one point but not parallel at other points. That is only possible in curved geometry.
So spacetime curvature is physically measurable. Curvature is tidal gravity. The “direction” of curvature around a large spherical object leads to stretching in the radial direction and compression in the tangential directions. If the object were composed of exotic matter then the opposite “direction” of curvature would imply compression in the radial direction and tension in the tangential directions.