"what I don't get is the vector pointing away from the wheel(the right hand rule). I spent a lot of time thinking but it just doesn't make sense."
It's fundamentally arbitrary. But it works.
Mathematicians had a way to describe things spinning in two dimensions, with complex numbers. If you have a wheel with a marker on it, you can describe the location of the marker with a complex number to describe its position compared to the center of the wheel. Then you can describe it moved to some other place on the circle by multiplying by another complex number that has length 1. You can rotate to the left or the right.
But in 3 dimensions it's more complicated, and anyway there are questions of mass and force. If you hit a spinning baseball, some of the force goes into changing its linear velocity and some into changing the spin. Lots of stuff to figure out. The first step is deciding how to measure spin.
You can measure the amount of spin by how fast the angles change. Then you want to measure the direction of spin in 3D. If it's a wheel that spins around an axle, you might want to measure the direction of the axle. And then there are two directions it can spin, right and left, just like the complex plane. Since it's completely arbitrary which of them you choose to write which way, they chose the Right Hand Rule. They could have used the Left Hand Rule just as easily, but they didn't. If you use the Left Hand Rule you'll confuse people.
There are lots of other ways to do it. For example you could have a vector to represent a radius around a center, and a second vector to represent the velocity of a point at that distance. The way it gets done generally works.
Except that in some circumstances you can calculate things in the wrong order and have part of the angular motion cancel out wrong and get "gimbal lock". This will not happen in 3D if you use 4D math instead. You can have the space part of the number show an axis, and if it is perpendicular to the orbit you want, it will act just like the complex-number case. But if you multiply the axis by something that isn't perpendicular, it will trace out an elliptical orbit and the time part will show how far ahead or behind in time it will be at that fraction of the cycle. Or you can do a more complicated operation to rotate or revolve a 3D structure as a unit.
This was first worked out as quaternions, but it has been repeated as part of geometric algebra, and it was repeated again as spinor theory.