The suggestion that the summation point for the net torque is that it is the center of mass is so the equations of motion to assume their simplest form. In general, the COM is not an inertial frame.
In order to understand where the idea of an inertial reference frame comes from, imagine a system of particles, each moving with $\vec{v}_i$ and having mass $m_i$, and describe the total linear and angular momentum.
First, require that each position $\vec{r}_i$ is decomposed into the location of the COM plus a relative component that remains a fixed distance from the COM
$$ \vec{r}_i = \vec{r}_C + \vec{d}_i $$
Because the above is described in a reference frame you can differentiate the above as
$$ \vec{v}_i = \vec{v}_C + \vec{\omega} \times \vec{d}_i = \vec{v}_C + \vec{\omega} \times (\vec{r}_i - \vec{r}_C) $$
and again
$$ \vec{a}_i = \vec{a}_C + \vec{\alpha} \times (\vec{r}_i - \vec{r}_C) + \vec{\omega} \times (\vec{v}_i - \vec{v}_C) $$
without considering the motion of the frame.
Now the total linear momentum is
$$ \vec{p} = \sum_i m_i \vec{v}_i = \sum_i m_i \vec{v}_C + \sum_i m_i \vec{\omega} \times (\vec{r}_i - \vec{r}_C) $$
Now consider that $\sum_i m_i \vec{v}_C = \vec{v}_C \left( \sum_i m_i \right) = m\,\vec{v}_C$, and that $\sum_i m_i \vec{r}_i = m\, \vec{r}_C $ by the definition of center of mass, the above is
$$\vec{p} = m \,\vec{v}_C $$
The total angular momentum about the (moving) center of mass is
$$ \vec{L}_C = \sum_i \vec{d}_i \times m_i \vec{v}_i = \sum_i \vec{d}_i \times m_i ( \vec{v}_C + \vec{\omega} \times \vec{d}_i ) $$
After further simplifications the above becomes
$$ \vec{L}_C = \sum_{i}m_{i}\vec{d}_{i}\times\left(\vec{\omega}\times\vec{d}_{i}\right)
= \mathrm{I}_C \vec{\omega}$$
As you can see it is the relative distance of each particle to the COM that counts here $\vec{d}_i = \vec{r}_i - \vec{r}_C$.
For dynamics you need the following definitions of force and torque
$$ \sum_i \vec{F}_i = \frac{\rm d}{{\rm d}t} \vec{p} $$
$$ \sum_i \left( \vec{\tau}_i + \vec{d}_i \times \vec{F}_i \right) = \frac{\rm d}{{\rm d}t} \vec{L}_C $$
again, by definition the center of mass is not an inertial frame location.