The point here is that, if $\omega\in SO(3)$, and if $\omega_1\cdot \omega_2=\omega \in SO(3)$ is the combination rule on abstract elements, then a representation (by matrices) $U$ is a map $\omega\mapsto U(\omega)$ so that the rule
$$
\omega_1\cdot \omega_2=\omega \quad \Rightarrow \quad U(\omega_1)\cdot U(\omega_2)=U(\omega) \tag{1}
$$
for any $\omega_1,\omega_2,\omega\in SO(3)$ is also satisfied by the matrices $U(\omega)$ representing the elements. There is a theorem stating that, for $SO(3)$ and a bunch of others, all representation are equivalent to unitary representations, so that $U(\omega^{-1})=U^{-1}(\omega)=U^\dagger(\omega)$.
Although the so-called defining representation is in terms of a $3$-dimensional space on which $SO(3)$ acts "naturally", there may be matrices of dimension other than $3$ that satisfy the basic composition law of $\omega_1\cdot \omega_2=\omega $ or its
matrix version of Eq.(1).
You can "obtain" a representation by larger matrices by tensoring and decomposing the resulting representation. For instance, if
$\{\vert {1}\rangle ,\vert {2}\rangle,\vert {3}\rangle\}$ are a basis for the $3$-dimensional irrep of $SO(3)$, then the set$\{\vert i\rangle\vert j\rangle\}$ spans a 9-dimensional space with
$$
U(\omega)\left[\vert i\rangle\otimes\vert j\rangle\right]:=
\left[U(\omega)\vert i\rangle\right]\otimes \left[U(\omega)\vert j\rangle\right]
$$
will provide you with a $9$-dimensional representation, which turns out to be reducible. Note that I'm abusing the notation here because on the left I have $U$ as a $9\times 9$ matrix but on the right the $U$'s are $3\times 3$ matrices.
In fact, the $9\times 9$ representation is reducible: it contains $L=2,1,0$, i.e. irreducible pieces of dimensions $5,3$ and $1$. The $L=2$ and $L=0$ irreps are spanned by symmetric combinations like $\vert 1\rangle\vert 2\rangle+ \vert 2\rangle\vert 1\rangle$ etc, while the $L=1$ contains antisymmetric pieces.
In cases other than $SO(3)$ (or $SU(2)$), one can also obtain inequivalent representations by taking the conjugate. The simplest example would be $SU(3)$, where the defining representation ($3\times 3$) is often denoted by $\textbf{3}$ or $(1,0)$ in the Dynkin scheme, and where its (non-equivalent) conjugate is denoted by $\textbf{3*}$ or $(0,1)$. One can then construct any representation by tensoring a suitable number of copies of $(1,0)$ and $(0,1)$ and decomposing the result.
Note that $SO(3)$ representations (and also $SU(2)$ representations) are "self-conjugate" in the sense that taking the conjugate yields the same representation.
