Yes, you are right. This is a more geometrical problem than anything else. If a point object were placed closed to the mirror, you can imagine two light rays reaching the mirror from there (above and below the principle axis). Nowm if you drag that point object all the way back (further away from the mirror), the two light rays tend to point in the same direction, because the distance to the mirror is getting larger than the aperture of the mirror itself. This means that light rays can be approximated as coming from a very infinity with non-zero angular width.
There is no proof for this that the object will be at infinity. Unless we assume that $u$ (the object distance) is a really large number. So, in the focal length expression, $1/f= 1/v + 1/u$
$$1/u \to 0$$ because $u>>f$.
Our day to day life mirror apertures (and hence smaller focal lengths) are small compared to the distances we see ourselves in real life.
EDIT: Whatever assumptions here work well whether or not the mirror or the lens suffers from chromatic abberations (if you are learning CBSE, you might come to that point towards the end - each color depending on their refractive indices - in the cases of lens only, fall off at different focal points. This is the region that the user Vincent was talking about). Rays coming from a distance greater than that of the aperture is considered the "infinity". This is the basic idea.