OP already seems to have thought long and hard about this and makes good points. In this answer we will review the argument for why constraint forces could be infinite.
We will assume that OP talks about holonomic$^1$ constraints. To be concrete, let the constraint be that some generalized coordinate vanishes $$q~\approx~0.$$
We can implement the constraint
ideally via a Lagrange multiplier term$^2$
$$L_1~=~L_0 +\lambda q,\tag{1}$$
where the constraint force can be identified with the Lagrange multiplier $\lambda$;
or pragmatically via a stiff spring potential
$$L_2~=~L_0 -\frac{k}{2} q^2,\tag{2}$$
where the spring constant $k$ is very large.
If $E$ denotes a characteristic energy available to the system, it is reasonable to expect
$$\frac{k}{2} q^2~\lesssim~ E,$$ or
$$|q|~\lesssim~{\cal O}(k^{-1/2}).$$
Hence the spring force
$$ |F|~=~|-kq|~\lesssim~{\cal O}(k^{1/2})~\to~\infty\quad{\rm for}\quad k~\to~\infty.$$
In other words, the spring force $F$ could be very large, and unbounded from above as $k\to\infty$. Of course, it might not be large at all the time. E.g. there could be an oscillatory pattern.
In particular, if one identifies the spring force $F$ in model 2 with the constraint force $\lambda$ in model 1, one can argue that the (absolute value of) the latter could be very large, cf. OP's title question.
We can unify model 1 & 2 via the Lagrangian
$$L_3~=~L_0 +\frac{\lambda^2}{2k} +\lambda q.\tag{3}$$
The EOM for $\lambda$ is
$$\lambda~\approx~ -kq.$$
On one hand
$$ \lim_{k\to\infty} L_3~=~L_1.$$
On the other hand, for finite $k>0$, if we integrate out/eliminate $\lambda$ from $L_3$ via its EOM, we exactly get $L_2$.
--
$^1$ Semi-holonomic constraints are quite subtle, cf. e.g. this Phys.SE post.
$^2$ The Lagrangian $L_0$ could in principle contain many degrees of freedom, i.e. depend on many generalized coordinates. We will assume that the system has at least 1 more coordinate than $q$, so that the system is non-trivial.