Rotations along different axes do not commute, but vector addition commutes, therefore, angular displacement cannot be a vector.
Angular displacements form a group however, the rotation group $SO(3)$, consisting of orthogonal matrices satisfying $O^{-1}=O^T$ and $\det O=1$.
If you consider a one-parameter family of rotations, indexed by, say, time, so a family $O_t$, such that $O_t\in SO(3)$ for all $t$ and such that $O_0=1$ the identity matrix, then the angular velocity at $t=0$ is given by $$ \omega(t=0)=\frac{d}{dt}O_t|_{t=0}. $$
To evaluate this derivative we consider that $O_tO_t^T=1$ for all $t$ and as such $$ \frac{d}{dt}(O_tO_t^T)|_{t=0}=\frac{d}{dt}O_t|_{t=0}+\frac{d}{dt}O_t^T|_{t=0}=0 $$ thus $$ \omega(0)+\omega^T(0)=0\Longrightarrow\omega(0)=-\omega^T(0), $$ so $\omega(0)$ is skew-symmetric. In the above derivation, during the product rule the second factor in the first term and the first factor in the second term got eliminated because both $O$ and $O^T$ are the identity matrix at $t=0$.
Now, orthogonal matrices form a group, because they are closed under matrix multiplication and inversion, but do not form a linear space, because they are not closed under addition and scalar multiplication.
Antisymmetric matrices on the other hand, are closed under addition and scalar multiplication, and are also closed under commutators, so they do form a vector space, and also a Lie-algebra. In fact, if you have learned any Lie theory, you know that given any Lie-group (which $SO(3)$ is), the Lie algebra of a group is generated by the derivatives of one-parameter subgroups of the Lie group at the identity.
Moreover, in 3 dimensions and only in 3 dimensions, antisymmetric matrices have three independent components, same amount as vectors, so they can be mapped to vectors. This map is given by $\omega_{ij}\mapsto\omega_i=\frac{1}{2}\sum_{j,k=1}^3\epsilon_{ijk}\omega_{jk}$ (the factor of 1/2 might not be needed, but am too lazy to check now).
You can also check that this map turns commutators into cross products, so if $A,B$ are antisymmetric matrices, and $\star$ is the map from antisymmetric matrices to vectors, then $$ \star[A,B]=\star A\times\star B $$ (an additional constant factor like the 1/2 before might enter the picture here, once again, am too lazy to check).
This is why angular velocity is a vector but angular displacements aren't. Angular displacements are elements of a Lie-group, which can be differentiated, but this derivative will leave the Lie-group and will become Lie algebra elements. And Lie algebra elements do form a vector space.
