I understand $SO(3)$ is the right group for proper rotation of its Hamiltonian. $O(3)$ describes both proper and improper rotations and the system has symmetry even for reflection. $O(3)$ is not connected and this makes harsh to study its Lie Algebra. For this reason we study proper and improper rotation apart.
We then notice $SO(3)$ is doubly connected, so it's easier to handle the universal covering $SU(2)$. From this I wonder: why not study $U(2)$, a compact and connected group without a restriction on matrixes determinants?
Does it describe improper rotation too, or does it add simmetries not described by the Hamiltonian?