As you state, the dipole will radiate EM radiation with power $P \propto \omega^{4} \sin ^{2} \chi$, with $P > 0$ if $\chi \neq 0$.
But as the radiated EM waves carry momentum $ p \propto E \times B$, so they also carry angular momentum $L = r \times p$, where $r$ is the distance from the axis of rotation ($E, B$ being the electric and magnetic fields). The rate of angular momentum loss can be written as (see Appendix B here)
$$ \dot{L}=\dot{\mathcal{E}} / \omega$$
where $\dot{\mathcal{E}}$ is the energy loss rate.
Adddition :
To clarify my earlier sloppy remark: As I read, the source of the magnetic field of a neutron star is typically associated with its crust as the core region is assumed to become a Type-I superconductor, which expells the magnetic field owing to the Meissner effect.
In this crust region, it is typically the electrons that can move relatively freely (e.g., due to circulation currents owing to von Zeipel theorem, but there may be more reasons).
These currents will produce a magnetic field, and if the entire system starts to rotate, then this results in a time-varying magnetic field, which results in EM radiation.
What causes the torque?
If the OP agrees that the radiation takes away angular momentum at the rate $ \dot{L}=\dot{\mathcal{E}} / \omega$, then the change in mass of the neutron star is $\delta M = \delta \mathcal{E} /c^2$ and change in angular momentum is $\delta L = \delta \mathcal{M}c^2 /\omega$. This implies a change in specific angular momentum $\ell
= c^2 /\omega$.
However, if we assume solid body rotation, then the maximum specific angular momentum of the matter in the star can be $\ell = I\omega / M \approx R^2\omega$, $I, M, R$ being the moment of inertia, mass and radius of the star.
As $c^2/\omega \gg R^2 \omega$, so the radiation takes away much more specific angular momentum than is carried by any part of the star. As such, the entire configuration has to slow down.
If the OP is after a mechanism using which the star slows down during the emission of radiation, then I think one would need to instead focus on the light-matter interaction problem and deal with it quantum mechanically. But qualitatively, the neutron star is slowing down due to redistribution of angular momentum where the radiation takes away a relatively large chunk with it.