This is a very complex question and I will try to clarify as much as I can.
You are saying that you are assuming for some reason that absorption leads to less momentum transfer then elastic scattering.
But think about it, when a photon gets absorbed by an atom/electron system, the photon ceases to exist. Momentum has to be conserved. Where does the momentum go? All the momentum of the photon gets transferred to the atom/electron system. Yes, the atom receives a recoil.
Gaining kinetic energy does not mean that it will gain thermal energy − kinetic energy is only thermal energy when it is in randomized directions. You can have objects which are very cold but moving very fast, ranging from ice cubes fired from a potato cannon all the way to atoms in particle accelerators with a high velocity but a thin velocity spread.
That said, the photon kick to the atom's center of mass does not mean that energy is somehow non-conserved. Instead, if the final state of motion (after absorption of the photon's momentum) has a higher kinetic energy than the state of motion before the photon absorption, this energy deficit is provided by the photon: in other words, the transition frequency gets blue-detuned, and the photon's energy needs to provide both for the change in internal energy and the change to the center of mass's kinetic energy.
Conservation of momentum in photon-atom collision
So the argument, that absorption leads to less momentum transfer then elastic scattering is not correct.
Now lets talk about heat. You are not saying it, but I assume you are only asking about visible wavelengths and surface heating. But please note that a material can be heated up by non-visible wavelengths and inelastic scattering deeper then the surface.
Now you are asking, why does a darker surface heat up faster then a brighter surface? Darker surfaces absorb (without re-emission) more visible wavelength photons and elastically scatter less photons. Brighter surfaces elastically scatter more visible wavelength photons and absorb (without re-emission) less. Please note that the surface will later re-emit the absorbed excess energy, because it is trying to reach thermal equilibrium with its environment.
Now the momentum of a photon is a vector (4-vector). When a photon gets absorbed by an atom/electron system, the photon ceases to exist and all its energy gets transferred to the atom/electron system. In the case of elastic scattering, the photon still exist, and keeps all its energy. The momentum vector only changes direction. The magnitude of the momentum vector is unchanged during elastic scattering.