The dimension of the Hilbert space in the stated setup is four, and thus all matrix representations of the operators should be of dimension four. This is independent of basis representation.
To wit: let's start with the spin-addition basis $|J, M\rangle$ where we have the $4$ states $|J=1, M=1\rangle$, $|J=1, M=0\rangle$, $|J=1, M=-1\rangle$ and $|J=0, M=0\rangle$, and we associate them with the basis column vectors in that order (that is $|J=1, M=1\rangle = (1, 0, 0, 0)^T$ etc.). Then
$$\vec{S}^2 = \hbar^2\begin{pmatrix}
2 & 0 & 0 & 0 \\
0 & 2 & 0 & 0 \\
0 & 0 & 2 & 0 \\
0 & 0 & 0 & 0
\end{pmatrix} $$
and
$$S_z^2 = \hbar^2\begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0
\end{pmatrix} $$
Now we can work in the separated basis with the four basis states $|m_1=\hbar/2, m_2=\hbar/2\rangle$, $|m_1=\hbar/2, m_2=-\hbar/2\rangle$, $|m_1=-\hbar/2, m_2=\hbar/2\rangle$, $|m_1=\hbar/2, m_2=-\hbar/2\rangle$ and we associate them again with the basis column vectors in that order. The matrix representation of $S_z^2$ is still diagonal
$$S_z^2 = \hbar^2\begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 1
\end{pmatrix} $$
but we get non-diagonal terms for the representation of $\vec{S}^2$. However calculating $\langle m_1, m_2 | \vec{S}^2 | m_1', m_2' \rangle$ is quite straight-forward, using CG. We have two states that can only have non-zero matrix elements with themselves (as they are also states in the momentum-added basis):
$$ \langle m_1=\uparrow, m_2=\uparrow | \vec{S}^2 | m_1', m_2' \rangle = 2\hbar^2\delta_{m_1', \uparrow}\delta_{m_2',\uparrow}$$
$$ \langle m_1=\downarrow, m_2=\downarrow | \vec{S}^2 | m_1', m_2' \rangle = 2\hbar^2\delta_{m_1', \downarrow}\delta_{m_2',\downarrow}$$
and have a non-diagonal matrix elements when we look at the subspace associated with total $S_z = 0$:
$$ \langle m_1=\uparrow, m_2=\downarrow | \vec{S}^2 | m_1=\uparrow, m_2=\downarrow \rangle = \langle m_1=\downarrow, m_2=\uparrow | \vec{S}^2 | m_1=\downarrow, m_2=\uparrow \rangle = \hbar^2$$
$$ \langle m_1=\hbar/2, m_2=\downarrow | \vec{S}^2 | m_1=\downarrow, m_2=\uparrow \rangle
= \hbar^2$$
so we end up getting
$$\vec{S}^2 = \hbar^2\begin{pmatrix}
2 & 0 & 0 & 0 \\
0 & 1 & 1 & 0 \\
0 & 1 & 1 & 0 \\
0 & 0 & 0 & 2
\end{pmatrix} $$
You can now convince yourself that the two representations are related to one another by basis transformations, which is given exactly by the CG coefficients (which is not surprising - this is how we got the matrices in the first place).