If I understand it correctly the laws of conservation linear and angular momentum and kinematic energy isn't enough since these are just three equations and four unknown values. Another thing is there is no way to clearly describe such a collision with only linear and angular velocities.
So what is the way to approach this? Which part of physics is focused on this topic?
The problem can be solved using the linear and angular velocities, and the conservation laws of energy, linear momentum, and angular momentum, but as you have noted, an extra condition needs to be applied. This is the specification of the direction in which the impulse between the two bodies acts, at the moment of contact.
The linear and angular velocities of both particles after collision may be related to those before collision, by equations involving the impulse. The impulse acts like an instantaneous force (on the centres of mass of the particles, changing the linear momenta) and, at the same time, as an instantaneous torque (changing the angular momenta about the centres of mass). Writing these equations down correctly, i.e. in accordance with Newton's third law, automatically guarantees conservation of linear and angular momentum. To give the flavour of these equations, here they are for linear rotors such as rods of mass $m$, which do not rotate about their "long" axes, but have a moment of inertia $I$ about the two perpendicular axes: \begin{align*} m\mathbf{v}_1' &= m\mathbf{v}_1 + \mathbf{C} & I\boldsymbol{\omega}_1' &= I\boldsymbol{\omega}_1 + (\mathbf{r}_c-\mathbf{r}_1)\times\mathbf{C} \\ m\mathbf{v}_2' &= m\mathbf{v}_2 - \mathbf{C} & I\boldsymbol{\omega}_2' &= I\boldsymbol{\omega}_2 - (\mathbf{r}_c-\mathbf{r}_2)\times\mathbf{C} \end{align*} Here the centres of mass are at $\mathbf{r}_1$ and $\mathbf{r}_2$ and the point of contact is at $\mathbf{r}_c$. Velocities $\mathbf{v}$ and angular velocities $\boldsymbol{\omega}$ after the collision are denoted by a prime $'$, and $\mathbf{C}$ is the (unknown) impulse vector.
As you can see, these are vector relations in three dimensions, so there are more unknowns (and more equations) than you mentioned in your question, but basically you are right: we need to specify something more about the collision in order to get a unique solution of the problem.
Typically, one adopts the "smooth collision" model, in which the impulse $\mathbf{C}$ is constrained to be normal to the surfaces of both the particles (at the moment of collision, the particles are touching, so they have a common normal). In other words, there is no component of the interaction parallel to the surfaces in contact. So, given the direction of $\mathbf{C}$, it only remains to compute its magnitude. This can be done if one imposes conservation of energy.
There is an alternative model, called the "rough collision" model, which allows the impulse to act in such a way as to reverse the relative velocities of the points on the colliding bodies which are in contact. The equations turn out slightly differently, but the basic scheme is the same.
Of course, one can also devise inelastic collision models, with a coefficient of restitution, in which the kinetic energy is not conserved.
One area of physics using these equations is the molecular dynamics of hard particles. The hard sphere model is an old and simple (but very useful) model of atomic interactions, and has evolved into nonspherical hard particle models of molecular systems. There is even a software package DynamO which implements these so-called event-driven particle simulations. [Disclaimer: I have no connection with the developers of DynamO.]