I'm working through Taylor's Mechanics and he seems to assume this implicitly at a few points. Intuitively, I agree that this should be the case. For while at any time during the rolling there ought to be a force of friction acting on the point of the object in contact with the surface, this force never "acts through a distance" since the object just rolls away. Trying to think more rigorously, I've come up with the following:
Thinking only in two dimensions for brevity's sake (although I see no reason why this argument can't be extended to three), and taking an arbitrary point on the object along its surface, the force of friction acting on this point is zero everywhere except for the point in time when it is in contact with the surface on which it is rolling. Assuming the path of the point can be given a continuous parameterization $(g_1(t),g_2(t))$ (I think this is a safe assumption), and taking an arbitrary partition of this curve $a = t_0<t_1...<tn=b$ and $c_i \in [t_{i-1}, t_i]$, each term in the approximating sums $\sum_{i=1}^{n}F_1(g(c_i))\cdot (g_1(t_i)-g_1(t_{i-1}))=S_1$ and $\sum_{i=1}^{n}F_2(g(c_i))\cdot (g_2(t_i)-g_2(t_{i-1}))=S_2$, where $(F_1, F_2)$ is a vector field defined on $A \supset g([a,b])$ representing the friction, is zero except in the case that $c_i$ is the point of contact of the point on the object and the surface. However since $g_1, g_2$ are continuous, for a sufficiently fine mesh of the partition, even if there is a $c_i$ corresponding to the point of contact in these sums, this term will be sufficiently small so as to make $|S_j-0| \lt \epsilon$ true for any $\epsilon \gt 0$, $j \in \{1,2\}$. Since these are the approximating sums of the Stieltjes integrals $\int (F_1\circ g)dg_1$ and $\int (F_2\circ g)dg_2$, $\int (F_1\circ g)dg_1 = \int (F_2\circ g)dg_2 = 0$. Whence we get $W_{fric} = \int_{g}\left\langle F, dx\right\rangle = \int_{g}F_1dx + \int_{g}F_2dy = 0$.
I am fairly confident in this argument, however I realize, as is often the case in physics, that there may be complicating factors that make a narrow mathematical argument like this untrue in general. So is this reasoning correct? If not, what is an argument for friction doing zero work on an object that is rolling without slipping? Or, is this not true at all?