Consider a volume charge distribution $\rho({\bf r'})$. The electric field at ${\bf r}$ is
$${\bf E}({\bf r})=\iiint\frac{1}{4\pi\epsilon_0}\frac{\rho({\bf r'})}{R^2}\hat{\bf R}\, \mathrm d^3{\bf r'}$$
where ${\bf R}={\bf r}-{\bf r'}$.
My question is at positions ${\bf r}$ where $\rho \ne 0$, the integrand blows up when ${\bf R}={\bf 0}$. But I know from EM textbooks that ${\bf E}({\bf r})$ at where $\rho({\bf r})\ne 0$ is well-defined.
Can one explain this mathematically?
Edit: Here is own thought, but I am not sure whether it's correct.
Consider this as a purely mathematics question first.
Then I think the question is similar to whether $$\int_0^1\frac{1}{\sqrt{x}}dx$$ is undefined or equals $2$.
In my opinion, strictly speaking, the above integral is undefined, because the integrand is undefined at $x=0$.
However, since $$\lim_{\delta\rightarrow0^+}\int_\delta^1\frac{1}{\sqrt{x}}dx=2$$ we usually just say $$\int_0^1\frac{1}{\sqrt{x}}dx=2$$
So in my opinion, when we write $${\bf E}({\bf r})=k\iiint\frac{\rho({\bf r'})}{|{\bf r}-{\bf r'}|^2}\hat{\bf R}d^3{\bf r'}$$ we are actually saying $${\bf E}({\bf r})=\lim_{\delta\rightarrow0^+}k\iiint_{D_{\delta}}\frac{\rho({\bf r'})}{|{\bf r}-{\bf r'}|^2}\hat{\bf R}d^3{\bf r'}$$ where $D_\delta$ is the space minus a ball with radius $\delta$ centered at ${\bf r}$.
It can be shown that if $\rho({\bf r})$ is finite or does not go to infinity $\textit{too fast}$, then the limit exists.
If the above is true, then the physics question one should ask is whether this gives the correct answer in physics.
In classical EM, the $\textit{macroscopic}$ field ${\bf E}$ is in fact the average of the $\textit{microscopic}$ field ${\bf e}$. (Landau and Lifshitz, $\textit{Electrodynamics of Continuous Media}$, p.1.)
So you can consider ${\bf E}({\bf r})$ as the average of ${\bf e}$ over a ball with radius $\delta$ centered at ${\bf r}$.
Now, it can be shown that
(1) the average field over the ball due to charges outside the ball is the same as the total field due to all charges outside the ball at the center of the ball, and
(2) the average field over the ball due to charges inside the ball is $$-k\frac{{\bf p}}{\delta^3}$$ where ${\bf p}$ is the total dipole moment inside the ball.
(Griffith, $\textit{Introduction to Electrodynamics}$, p. 156-157, problem 3.41).
So when $\delta$ is small macroscopically, ${\bf p}\rightarrow{\bf 0}$, and we have $${\bf E}({\bf r})=k\iiint_{D_{\delta}}\frac{\rho({\bf r'})}{|{\bf r}-{\bf r'}|^2}\hat{\bf R}d^3{\bf r'}$$