In quantum mechanics, we implement transformations by operators $U$ that map the state $|\psi\rangle$ to the state $U|\psi\rangle$. Alternatively, we could transfer the action of $U$ onto our operators: $$ O\mapsto U^\dagger OU $$ These operators $U$ ought to constitute a representation of the transformation group of interest. The question we then ask is: which representation shall we choose? I have learnt that we pin down the matrices $U$ that we desire by asking that the position operator, or the position eigenstate, transforms in the appropriately geometric way: if $R$ is some rotation matrix in 3D, then we wish that $$ U(R)^\dagger \vec{x} U(R) = R\vec{x} $$
I have multiple questions regarding this procedure:
1) It is often argued that $U$ is unitary since 'probabilities must be conserved'. But it is clear from my studies that operators representing Lorentz boosts are not unitary. How are these two facts reconciled?
2) On a related note, we say that a transformation is a symmetry of the system if the Hamiltonian is preserved under the transformation. This makes sense to me in that it is the Hamiltonian that defines the system; if the Hamiltonian is unchanged, then the old solutions to Schrodinger's equation are still solutions. However, it is clear that whatever transformation operator implements a Galilean boost (in non-relativistic QM) or Lorentz boost (in relativistic QFT) is not going to leave the Hamiltonian unchanged. That doesn't mean that boosts aren't symmetries of our system, however. What's going on here? Why exactly is $U^\dagger H U = H$ the condition for $U$ to be a 'symmetry'?
3) Demanding that the operators $U$ satisfy the property (choosing translation for concreteness) $$ U^\dagger \vec{x} U = \vec{x} + \vec{a}$$ doesn't immediately tell us how the transformation will affect other operators, such as the momentum or angular momentum operators. For the case of a translation by $a$, we find that $$ U = \exp\left( -i\frac{\vec{a}\cdot\vec{p}}{\hbar}\right) $$ satisfies the above equation (work infinitesimally). This then tells us that $U^\dagger \vec{p} U = \vec{p}$. This is what we want --- translations in space do not affect the momentum. But we haven't told our operators $U$ that they ought to satisfy this property, so the fact that they do seems something of a coincidence. That is, requiring only that conjugating the position operator with $U(R)$ rotates it seems to force every vector operator to be rotated under conjugation with $U(R)$. Is there something fundamental underlying this?
4) In QFT, we wish to implement Lorentz boosts with operators $$ S = \exp \left( -i\frac{\omega_{\mu \nu} M^{\mu \nu}}{2\hbar}\right) $$ where $\delta^\mu_\nu + \omega^\mu{}_\nu$ is the infinitesimal Lorentz transformation we are representing. Now for $T$ representing a different, not necessarily infinitesimal, Lorentz transformation $\Lambda$, we have $$T^{-1} S T = T^{-1} \left(1 - i\frac{\omega_{\mu \nu} M^{\mu \nu}}{2\hbar} \right) T$$ But the condition that the matrices $S$ and $T$ must constitute a representation of the Lorentz group implies that the combination on the left hand side ought to equal $$\exp \left( -i\frac{\Omega_{\mu \nu} M^{\mu \nu}}{2\hbar}\right) $$ where $\Omega_{\mu \nu}$ defines the composite transformation: $$ \delta^\mu_\nu + \Omega^\mu{}_\nu \equiv (\Lambda^{-1})^\mu{}_\rho(\delta^\rho_\sigma + \omega^\rho{}_\sigma)\Lambda^\sigma{}_\nu $$ Comparing these equations then gives $$T^{-1} M^{\mu \nu} T = \Lambda^\mu{}_\rho \Lambda^\nu{}_\sigma M^{\nu \sigma}$$ using the raising and lowering properties of the Minkowski metric. My question is similar to that in 3) --- the object $M^{\mu \nu}$ is just a collection of 6 operators, and it is not at all obvious that those indices ought to be tensor indices. And yet somehow --- just by asking for the matrices $S$ to constitute a representation --- we've managed to reproduce the tensor transformation law (admittedly we have a $T^{-1}$ here rather than a $T^\dagger$ --- this relates to my first question). In other words, we've got the correct geometric action of the Lorentz transformation on the operators $M^{\mu \nu}$, even though we never asked for it!
As you can tell, I'm very confused by the whole situation. I apologise if the above discussion sounds a bit muddled, and I would appreciate any help that can be given!