For large polycrystaline ferromagnets, the magnetization response to a high strength external magnetic field is usually limited by eddy currents. For magnets that are very small or have very low electrical conductivity, the reversal depends on how fast the magnetic domains can change. Since the question is about reversing the magnetization of a ferromagnet, we'll only consider large applied fields greater than the coercive field of the material.
Eddy Current Time Constant
Most common ferromagnetic materials are electrically conductive, so eddy currents are produced within the ferromagnet when the applied external magnetic field changes.
The $\tau_\mathrm{eddy} \sim L/R$ decay time constant of these eddy currents limits the rate of change of the magnetization. The precise value of $\tau_\mathrm{eddy}$ depends on the geometry, but roughly speaking we expect:
$$L \sim \mu \frac{A}{\ell} \qquad\qquad R \sim \rho \frac{\ell}{A}$$
so
$$\tau_\mathrm{eddy} \sim \frac{\mu}{\rho} \left(\frac {A}{\ell}\right)^2$$
where $\mu$, $\rho$, $A$, and $\ell$ are the ferromagnet's magnetic permeability, electrical resistivity, cross-sectional area, and length. Assuming the ferromagnet's size scale is $d$, then $A\sim d^2$ and $\ell \sim d$, so
$$\tau_\mathrm{eddy} \sim \frac{\mu}{\rho} d^2$$
Essentially the same limit can be derived from the frequency dependence of the skin depth, $\delta=\sqrt{2\rho/\omega \mu}$, since the skin depth and eddy current time constant both depend on the same physics.
The permeability of ferromagnetic materials depends on the magnetic field /strength, increasing up to some maximum value and then decreasing, but we can use $\mu_{\mathrm{max}}$ to make rough estimates of the time constant. For example, for $d\sim 1\,\mathrm{cm}$, the response times should be of order:
- $\tau_\mathrm{eddy} \sim 1$ second for an electrical steel ferromagnet with $\mu\approx 4000 \mu_0$ and $\rho\approx 4.72\times 10^{-7}\,\Omega \,\mathrm{m}$
Much faster response times can be achieved with other magnetic materials such as ferrites or Neodymium magnets:
- $\tau_\mathrm{eddy} \sim 80$ nanoseconds for MnZn ferrite with $\mu\sim 2500 \mu_0$ and $\rho\sim 4\,\Omega \,\mathrm{m}$
- $\tau_\mathrm{eddy} \sim 80$ microseconds for a $\mathrm{Nd_2}\mathrm{Fe_{14}}\mathrm{B}$ Neodynium magnet with $\mu\sim \mu_0$ and $\rho\sim 1.5\times 10^{-6}\,\Omega \,\mathrm{m}$
Domain Reversals
Eddy currents limit the magnetization reversal speed of the macroscopic magnets I believe you are asking about, but domain realignment is more important at the nanoscale.
A ferromagnetic material consists of many small magnetic domains. Within a domain, quantum effects align all the molecular magnetic dipoles. When the material is placed in a magnetic field, the domains more aligned with the field tend to grow and anti-aligned domains shrink, and when the field is large enough the orientation of the domains rotates to align with the field.
As Testina has noted in their answer, if a magnetic field $H$ is applied instantaneously, the time-scale for an ideal ferromagnet to flip its magnetization should follow the Arrhenius-Néel-Brown law (e.g. Eq. 2 of this paper):
$$\tau_{ANB}=t_0\,\mathrm{exp}\left(\frac{E_a - MV\mu_0 H}{kT}\right)$$
here $E_a$ is the activation energy aligning the spins to each other, $MV$ is the typical magnetic moment (Magnetization $\times$ Volume) of the domains, and $T$ is the temperature. The attempt time $t_0$ characteristic of a material can crudely be thought of as the average time between "attempts" for the domain to transition. It is bounded from below by the thermal time scale $h/kT$ ($\sim 2\times 10^{-13}\,\mathrm{s}$, for $T\sim300\,\mathrm{K}$), but is more typically $10^{-10}-10^{-9}\,\mathrm{s}$.
This equation tells us that if the magnetic field is strong enough, the transition time should become exponentially small (at least down to some limit). Simulations of tiny $10\,\mathrm{nm}$ neodynium boron magnets show the relaxation time falling from from $1$ to $10^{-10}$ seconds as the field increases past the coercive field from $3$ to $4$ Tesla, and then slowly decreasing to $\sim 10^{-12}\,\mathrm{s}$. Experimentally, there is a very large amount of research on flipping nanoscale domains, but instead of a solenoid, the magnetic fields are provided by fast laser pulses.