Your state (3.8.18) is a fully-symmetrized tensor (Kronecker) product of 2j oscillators, or spin doublets (spin 1/2s) arrayed to yield a spin j object, in this ingenious Jordan (Schwinger) construction.
So, by construction, (recalling this),
$$
\bbox[yellow]{e^{-i\beta J_y/\hbar} = e^{-i\beta j_y/\hbar} \otimes e^{-i\beta j_y/\hbar} \otimes ...\otimes e^{-i\beta j_y/\hbar} },
$$
where $j_y=\sigma_y/2$ for each tensor factor. That is to say, each of the 2j tensor factors only acts on its doublet/oscillator subspace and ignores all others. I am skipping the vacuum, a singlet, since it is rotationally invariant; also in this language.
So, sandwiching the product of oscillators in (3.8.18) by this rotation operator on the left and its inverse on the right, amounts to
$$
e^{-i\beta j_y/\hbar} a_+^\dagger e^{i\beta j_y/\hbar} \otimes e^{-i\beta j_y/\hbar} a_+^\dagger e^{i\beta j_y/\hbar} \otimes e^{-i\beta j_y/\hbar} a_+^\dagger e^{i\beta j_y/\hbar} \otimes ... \otimes e^{-i\beta j_y/\hbar} a_-^\dagger e^{i\beta j_y/\hbar}
$$
a total of 2j tensor factors, which acts on the vacuum. This amounts to each tensor factor transforming as
$$
a_+^\dagger \mapsto e^{-i\beta j_y/\hbar} a_+^\dagger e^{i\beta j_y/\hbar} = a_+^\dagger \cos(\beta/2) + a_-^\dagger \sin(\beta/2) \\
a_-^\dagger \mapsto e^{-i\beta j_y/\hbar} a_-^\dagger e^{i\beta j_y/\hbar} = a_-^\dagger \cos(\beta/2) - a_+^\dagger \sin(\beta/2) ,
$$
by the well-known reduction of Pauli vector exponentials. This is the expression following (3.8.20) in your display.
To test-drive this with a simple tractable example, consider j=1,
$$
|1,0\rangle= a_+^\dagger a_-^\dagger |0\rangle,
$$
so that, acting on the left with this rotation yields
$$
\Bigl (e^{-i\beta j_y/\hbar} \otimes e^{-i\beta j_y/\hbar} \Bigr ) \Bigl (a_+^\dagger \otimes a_-^\dagger\Bigr )|0\rangle \\
=\Bigl (a_+^\dagger \cos(\beta/2) + a_-^\dagger \sin(\beta/2) \Bigr )\Bigl ( a_-^\dagger \cos(\beta/2) - a_+^\dagger \sin(\beta/2) \Bigr ) |0\rangle \\
= \bigl (\sin \beta ~ ((a_-^\dagger)^2 -(a_+^\dagger) ^2 )/2 + \cos\beta ~ a_+^\dagger a_-^\dagger\bigr )|0\rangle \\
= \cos\beta ~|1,0\rangle + \frac{\sin\beta }{\sqrt 2}|1,-1\rangle - \frac{\sin\beta }{\sqrt 2}|1,1\rangle ,
$$
the coefficients yielding the $d^1_{0,m}$s.
NB If you really wish to eschew the above symmetric tensor product structure, simply recall that, by Schwinger's definition,
$$
\bbox[yellow]{e^{-i\beta J_y/\hbar}= e^{-{\beta\over 2} (J_+-J_-)/\hbar}
\equiv e^{\frac{\beta}{2} (a_-^\dagger a_+ - a_+^\dagger a_-)} },
$$
so you braid this operator past each of your 2j oscillators, all the way to the right where it trivializes to 1 operating on the vacuum. You will, of course, find the same result provided above!
$$
e^{\frac{\beta}{2} (a_-^\dagger a_+ - a_+^\dagger a_-)} a_+^\dagger e^{-\frac{\beta}{2} (a_-^\dagger a_+ - a_+^\dagger a_-)} = a_+^\dagger \cos(\beta/2) + a_-^\dagger \sin(\beta/2) ,
$$
and the orthogonal form for $a_-^\dagger$.
