I did some research and calculations:
To summarize: The relativistic rocket won't break apart, uniform acceleration along it is possible. But the observers will measure different accelerations due to the gravitational time dilation.
In more detail:
Let's assume the observer at the bottom measures $\alpha$ acceleration. So for an inertial observer outside (who draws the Minkowski chart) this accelerating observer's motion will be hyperbolic. The semi mayor axis of this hyperbola will be $c^2/\alpha$.
Let's say the length of the relativistic rocket is $h$ this is measured before the launch, and the mechanical stresses during the travel try to keep this value constant in the reference frame of the rocket, otherwise the rocket would break apart (we assume it don't break apart). As the rocket accelerates the plane of simultaneity rotates from the viewpoint of the inertial observer. So the two ends of the rocket won't trace two identical hyperbolas. But the two ends always connect two points on the two hyperbolas whose slope is the same (you can see this on the Rindler chart). So all parts of the rocket travel with the same speed at the local frame simultaneously, so the rocket's acceleration will be uniform and it won't break apart.
But the two hyperbolas are different. The bottom traces a hyperbola whose semi-mayor axis is $c^2/\alpha$, the top traces a hyperbola whose semi-mayor axis is $c^2/\alpha + h$. The acceleration that corresponds to the second hyperbola is $1 + \alpha h / c^2$ times smaller than $\alpha$.
This a bit paradoxical situation, because I stated that the acceleration is uniform along the rocket, now I state it's different due to the different hyperbolas.
This paradox can be resolved if I introduce gravitational time dilation, so I assume that the clock of the observer at the top ages faster with the rate I mentioned above. So the top observer measures less acceleration this way.
There is an event horizon at the Rindler-horizon where $h = -c^2/\alpha$, there the time stands still. This is somewhat analogous with the black hole's event ho+rizon.
The gravitational time dilation formula mentioned in the Wikipedia and the one mentioned in the comment is the same, but that exponentional formula never reaches zero, that would mean the Rindler-horizon does not exist... Which would be a bit odd. So I still need some research.
Update: The Wikipedia article has been fixed since last update. So the general formula to the gravitational time dilation is $e^{\int^h_0 g(h)dh / c^2}$, where $g(h)$ is the measured gravitational acceleration at the given level. For Rindler observers $g(h) = c^2/(H+h)$ where $H = c^2/\alpha$. Doing the integral gives $e^{ln(H+h) - ln(H)} = (H+h)/H$. Substituting $H$ back I will get the original $1 + \alpha h / c^2$ I mentioned earlier.