Mr. Zee seems to be saying that, since we shall define the array $\vec{p}$ as a vector, we have
$$\begin{align}
p_1'&=R_{11}p_1+R_{12}p_2\\
p_2'&=R_{21}p_1+R_{22}p_2\\
\end{align}$$
Let us define a new array (not vector) $\vec q$ as
$$\vec q \equiv \begin{pmatrix} ap_1\\ bp_2 \end{pmatrix}$$
If $\vec q$ as an array transforms naively under the transformation $R$, we get
$$\begin{align}
q_1'&=ap_1'\\
&=R_{11}ap_1+\frac{a}{b}R_{12}bp_2\\
&=R_{11}q_1+\frac{a}{b}R_{12}q_2\\
q_2'&=bp_2'\\
&=\frac{b}{a}R_{21}ap_1+R_{22}bp_2\\
&=\frac{b}{a}R_{21}q_1+R_{22}q_2\\
\end{align}$$
On the other hand, if $\vec q$ is indeed a vector, it must transform under $R$ as simply
$$\begin{align}
q_1'&=R_{11}q_1+R_{12}q_2\\
q_2'&=R_{21}q_1+R_{22}q_2\\
\end{align}$$
If these two transformation rules — the "naive" and "true" ones — are to be the same, then we must have $a=b$. However if $\vec q$ is simply defined to be a vector from the start, and therefore transform under the second transformation rule, then there is no such restriction necessary.
Ultimately I do not see the point in this at all.