Your expression "the angular momentum is $m_j \hbar$" (where $\hbar = h/2 \pi$) is incorrect. This quantity is the projection of the angular momentum on the $z$-axis; it represents the direction of the spin. This is why it corresponds to the $m$ quantum number, not the $\ell$ quantum number in the $| n \ell m \rangle$ basis.
Which angular momentum to use depends on what you're looking for. If you want simply the magnetic dipole moment, you use
$$
\mathbf{\mu} = \frac{-e}{2 m} \mathbf{L}
$$
Here $\mathbf{L}$ is the total angular momentum operator. Its full expression is a vector, but the eigenvectors of its square are $\hat{L}^2 | n \ell m \rangle = \hbar^2 \ell( \ell + 1) | n \ell m \rangle$. However, it's very possible that the problem you're doing wants some sort of energy due to the interaction of the dipole with a magnetic field, which is given by $\mathbf{ \mu \cdot B}$. If the magnetic field is taken to be in the $z$-direction, as it often is, this reduces to:
$$
E =\frac{-e}{2m} B L_z
$$
Now what you want is indeed the projection on the $z$-axis, and the operator $L_z$ does have the eigenvalues you mention, $\hat{L}_z | n \ell m \rangle = m \hbar | n \ell m \rangle$.