The way I see it is this : consider operators valued in the tensor product of the Hilbert space of states and some vector space, which transform in the right way under rotations (or more generally, under the symmetry transformations of your system). I don't know if this done properly in some textbook.
Let $\mathcal H$ be the Hilbert space of quantum states. We assume that it represents some system of particles in euclidean space $E_3$ (For simplicity, we consider only integer spin particles). For every rotation $R:E_3 \to E_3$, there is a corresponding unitary operator $U(R) : \mathcal H \to \mathcal H$, such that if $R,R'$ are two rotations, then :
$$U(RR') = U(R)U(R')$$
Then, the position operator $\hat{X}$ is a map $\mathcal H\to E_3\otimes \mathcal H$, which we can define by :
$$\forall x \in E_3, \hat X |x\rangle = x\otimes |x\rangle$$
For this collection of operators to truly be a vector, we need it to correctly transform under rotations. For any $x \in E_3$ and a rotation $R$ we have $U(R)|x\rangle=|Rx\rangle $ and therefore :
\begin{align}
\left(\mathbb I_{E_3}\otimes U(R)^\dagger\right)\hat XU(R)|x\rangle &= \left(\mathbb I_{E_3}\otimes U(R)^\dagger\right)\hat X|Rx\rangle \\
&= \left(\mathbb I_{E_3}\otimes U(R)^\dagger\right)(Rx\otimes |Rx\rangle) \\
&= Rx \otimes |x\rangle \\
&= \left(R\otimes \mathbb I_{\mathcal H}\right)\hat X|x\rangle
\end{align}
and therefore :
$$\left(\mathbb I_{E_3}\otimes U(R)^\dagger\right)\hat X U(R) = \left(R\otimes \mathbb I_{\mathcal H}\right)\hat X$$
In the same way, we define a vector value momentum operator $\hat P : \mathcal H\to E_3 \otimes \mathcal H$. We can then give a basis invariant formulation of :
$$[X^i, P^j] = i\hbar \delta^{ij}$$
Likewise, the angular momentum $\hat L = \frac{1}{m}\hat X\times \hat P$ can be defined in a basis invariant way from $\hat X$ and $\hat P$
Edit How to make sense of the fact that $\hat X$ is self adjoint ? Since the components of $\hat X$ in any basis are self-adjoint, we see that for any linear map $\ell :E_3 \to \mathbb R$, the operator $(\ell \otimes \mathbb I_{\mathcal H})\hat X : \mathcal H\to\mathcal H$ is self-adjoint.
Actually, since $E_3$ has a scalar product, we can directly define the adjoint of $\hat X$ as a map $\hat X^\dagger : E_3 \otimes \mathcal H \to \mathcal H$.
Then, if $\ell \in E_3^*$, we have :
\begin{align}
(\ell \otimes \mathbb I_{\mathcal H})\hat X &= \left((\ell \otimes \mathbb I_{\mathcal H})\hat X \right)^\dagger \\
&= \hat X^\dagger (\ell \otimes \mathbb I_{\mathcal H})^\dagger \\
&= \hat X^\dagger (\ell^\dagger \otimes \mathbb I_{\mathcal H}) \\
\end{align}
where $\ell^\dagger$ is the unique vector in $E_3$ such that :
$$\forall x \in E_3, (\ell^\dagger,x) = \ell(x)$$