It is well known that the Lorentz algebra can be written as two $SU(2)$ algebras. By defining
$$N_i=\frac{1}{2}(J_i+iK_i), \qquad N^{\dagger}_i=\frac{1}{2}(J_i-iK_i)$$
we have
$[N_i,N_j]=i\epsilon_{ijk}N_k,\qquad [N^{\dagger}_i,N^{\dagger}_j]=i\epsilon_{ijk}N^{\dagger}_k, \qquad[N_i,N^{\dagger}_j]=0$
We can find representations in the same way as the spin angular momentum operators of non-relativistic QM, with representations labeled by half-integers. The $(\frac{1}{2},0)$ representation corresponds to left-handed spinors. Similarly we could have a representation $(1,\frac{1}{2})$.
I'm a bit confused by this. In this scenario $N_3$ is a 3x3 matrix and $N^\dagger_3$ is a 2x2 matrix, then $J_3=N_3+N^\dagger_3$ which is a 3x3 matrix + a 2x2 matrix which doesnt make any sense.
Even the $(\frac{1}{2},0)$ representation could be seen as $J_3$ being a 2x2 matrix plus a scalar.
Where am I going wrong?