To expand on @Alfred's answer,
The gravitational attraction from the Earth as you dig down into the Earth is due only to the mass of the Earth enclosed by the sphere defined by the radius that you are at (see the derivation of Gauss's theorem for a proof of this). Therefore, as you go further down into the Earth, the amount of mass that is pulling you towards the center decreases, and the form of the equation that you cited changes.
Outside the earth (such that $R > R_E$), as you said, the gravitational acceleration is (classically),
$$g = \frac{G M}{R^2}$$
But once you enter the earth (at a radius such that $R < R_E$), the mass that is gravitationally attracting you gets smaller, according to the ration of the volume of the sphere inside your radius, and the sphere inside the Earth's radius$^1$.
$$M(R) = M \frac{\frac{4}{3} \pi R^3}{\frac{4}{3} \pi R_E^3}$$
$$M(R) = M \left(\frac{R}{R_E}\right)^3$$
Inserting that into the equation above, we find that the acceleration for $R < R_E$ is
$$g = \frac{G}{R^2} M \left(\frac{R}{R_E}\right)^3$$
Simplifying that gives us
$$g = \frac{G M}{R_E^3} R$$
Since the first part of that term is a constant, we can say $\alpha = \frac{G M}{R_E^3}$, and
$$g = \alpha R$$
This is a simple linear equation (see the line part of the graph in @Alfred's answer), which goes to zero, not infinity, as $R$ goes to zero.
$^1$*Note that this is a simplification based on the Earth having uniform density. That is definitely not the case, but since the amount of mass inside a smaller radius is smaller regardless, it will serve our purpose well of determining the limiting behavior of gravitational acceleration as R goes to zero.*