How do point particles transfer angular momentum between each other?
They don't, because there are no point particles. That's a mathematical myth. Unfortunately it is promoted by some seemingly authoritative sources.
I know that quantum physics says that one can't change the magnitude of spin of a point particle
Can you give me a reference for that? Only it's quantum field theory, not quantum point-particle theory. An electron is not some pointlike thing that has a field, it is field. In QFT it's described as an excitation of the electron field. In atomic orbital electrons "exist as standing waves". We can diffract electrons. An electron has a magnetic moment, the Einstein-de Haas effect demonstrates that "spin angular momentum is indeed of the same nature as the angular momentum of rotating bodies as conceived in classical mechanics". Saying the electron is a point particle is the electromagnetic equivalent of hanging out of a helicopter probing a whirlpool with a barge-pole, then when you can't feel anything solid, claiming that the solid thing in the middle must be very very small.
but that still leaves the question of how one changes the direction of spin.
A spin ½ particle doesn't really have a direction of spin. To appreciate this, imagine a disk rotating clockwise. Now walk round to the back of it, and note that you'd now say it was rotating anticlockwise. Then I spin it like a coin, so it's got two orthogonal rotations. Which way is it spinning? Every which way. But do note that I could have spun it like a coin with the other hand. There are two every which ways.
One possible way point particles can exchange angular momentum is by electromagnetism but then how do neutrally charged point particles exchange angular momentum?
There are no neutral point particles. A photon has no charge, but it has wave nature wherein E=hc/λ. As for massive neutral particles, have a look at the Wikipedia article on neutron magnetic moment: "The existence of the neutron's magnetic moment indicates the neutron is not an elementary particle. For an elementary particle to have an intrinsic magnetic moment, it must have both spin and electric charge". Also look at the neutron article and see structure and geometry of charge distribution along with free neutron decay and neutron diffraction. It's the wave nature of matter, not the point-particle nature of matter.
As electron does not seem to have any imperfections in its surface to grab and push to rotate. There does not seem to be any parts sticking out for which one can apply torque to.
That's true. It's spherically symmetric. I attempted to describe it here. As you can see from the downvotes, the people who tell you the electron is a point particle and "spin is intrinsic" didn't like it. But all the downvotes in the world won't make that hard scientific evidence go away. Make sure you read Goudsmit on the discovery of electron spin:
'When the day came I had to tell Uhlenbeck about the Pauli principle - of course using my own quantum numbers - then he said to me: "But don't you see what this implies? It means that there is a fourth degree of freedom for the electron. It means that the electron has a spin, that it rotates".'
There is an odd little non-sequitur that has crept into physics. You can see it in this old Stern-Gerlach article on Wikipedia:
"If this value arises as a result of the particles rotating the way a planet rotates, then the individual particles would have to be spinning impossibly fast. Even if the electron radius were as large as 2.8 fm (the classical electron radius), its surface would have to be rotating at 2.3×10$^{11}$m/s. The speed of rotation at the surface would be in excess of the speed of light, 2.998×10$^8$m/s, and is thus impossible."
Look closely and you'll see the sleight-of-hand. It says the electron can't be rotating like a planet, so it can't be rotating at all. That's wrong. Magnetic moment says its wrong. The Einstein-de Haas effect says its wrong. Electron motion in a magnetic field says its wrong. Of course it isn't rotating like a planet, it's a spin ½ particle. It's a spinor, with two orthogonal rotations. It rotates this way AND it rotates that way, and the AND acts as a multiplier.