Your observations are correct, and they rely on the premise that $\boldsymbol{p}=0$. This means that for all observers (summation point) the body is purely rotating without any translation.
The conclusion drawn is the for a purely rotating body the angular momentum measured is the same regardless of where the summation point is.
Now angular momentum is also the moment of momentum of a point particle, such as $\boldsymbol{L} = \boldsymbol{r}_C \times \boldsymbol{p}$, as well as for a non rotating, translating rigid body.
For the general case of a rotating and translating rigid body, angular momentum is the combination of the moment of momentum term and the intrinsic angular momentum $\boldsymbol{L}_C = \mathrm{I}_C\, \boldsymbol{\omega}$ which is what you call the "internal" angular momentum.
$$ \boldsymbol{L} = \mathrm{I}_C\, \boldsymbol{\omega} + \boldsymbol{r}_C \times \boldsymbol{p} \tag{1}$$
Here $\boldsymbol{r}_C$ is the location of the center of mass relative to the summation point
Obviously, there are cases like you pointed out where $\boldsymbol{L} = \mathrm{I}_C \,\boldsymbol{\omega}$ regardless of the summation point. If the body is not translating, if the center of mass is on the summation point and if the body is translating radially outwards from the summation point.
The more interesting question here is what does (1) tell us if we were to treat the rotating-translating rigid body as an equivalent particle, moving at some distance $\boldsymbol{r}_X$ with some momentum $\boldsymbol{p}$.
This exists in the form of a line that goes through a point $\boldsymbol{r}_X$ and is parallel to momentum $\boldsymbol{p}$. I will show that angular momentum resolved anywhere along this line is strictly parallel to momentum with a proportionality factor $h$ (called the pitch) such that $\boldsymbol{L}_X = h \,\boldsymbol{p}$.
This line in space is often called the axis of percussion.
What I am going to prove here is that there always is a point $\boldsymbol{r}_X$ and pitch $h$ which would allow us to decompose the general angular momentum vector $\boldsymbol{L}$ as follows
$$ \boldsymbol{L} = h\,\boldsymbol{p} + \boldsymbol{r}_X \times \boldsymbol{p} \tag{2}$$
To find these quantities take the dot and cross product of the two momentum vectors
$$\require{cancel} \left.\begin{aligned}\boldsymbol{p}\cdot\boldsymbol{L} & =h\,\left(\boldsymbol{p}\cdot\boldsymbol{p}\right)\\
\boldsymbol{p}\times\boldsymbol{L} & =\boldsymbol{p}\times\left(\boldsymbol{r}_{X}\times\boldsymbol{p}\right)\\
& =\boldsymbol{r}_{X}\left(\boldsymbol{p}\cdot\boldsymbol{p}\right)-\boldsymbol{p}\left(\cancel{\boldsymbol{r}_{X}\cdot\boldsymbol{p}}\right)
\end{aligned}
\right\} \begin{aligned}h & =\frac{\boldsymbol{p}\cdot\boldsymbol{L}}{\|\boldsymbol{p}\|^{2}}\\
\boldsymbol{r}_{X} & =\frac{\boldsymbol{p}\times\boldsymbol{L}}{\|\boldsymbol{p}\|^{2}}
\end{aligned} \tag{3}$$
Here $\boldsymbol{r}_X \cdot \boldsymbol{p} = 0$ because there is at least one point along the axis of percussion that is closest to the origin making $\boldsymbol{r}_X$ perpendicular to the line.
You can plug those back into (2) and see how the equations check out.
So any moving rigid body has a line in space whose a particle with the same momentum $\boldsymbol{p}$ exhibits an equivalent (or maybe equipollent is a better term) angular momentum.
There is a beautiful relationship between the location of the center of mass and the location of the axis of percussion. Without loss of generality place the origin on the rotation axis, with z-axis along the rotation and x-axis towards the center of mass

If the body has a radius of gyration $R_{\rm gyr}$ for the rotation axis $\boldsymbol{\omega}$ about the center of mass then the distance to the percussion axis $\| \boldsymbol{\ell} \|$ is described by the following equation
$$ \| \boldsymbol{c} \| \| \boldsymbol{\ell} \| = R_{\rm gyr}^2 $$
Note that $\boldsymbol{\ell}$ is always on the opposite side of the center of mass. Also, there are two special cases
- $\boldsymbol{c}=0$, body rotating about the center of mass, means that $\boldsymbol{\ell} = \infty$
- $\boldsymbol{\ell} =0$, body purely translating, means that $\boldsymbol{c} = \infty$ or the rotation center is at infinity.
In the scenario above you have
$$\begin{aligned}\boldsymbol{r}_{C} & =\boldsymbol{\hat{x}}c\\
\boldsymbol{\omega} & =\boldsymbol{\hat{z}}\,\omega\\
\boldsymbol{\omega}\times\boldsymbol{r}_{C} & =\omega \,c\,\boldsymbol{\hat{y}}
\end{aligned}$$
and
$$\begin{aligned}\boldsymbol{p} & =m\left(\boldsymbol{\omega}\times\boldsymbol{r}_{C}\right)=m\omega c\left(\boldsymbol{\hat{z}}\times\boldsymbol{\hat{x}}\right)=m\omega c\boldsymbol{\hat{y}}\\
\boldsymbol{L} & =\mathrm{I}_{C}\boldsymbol{\omega}+\boldsymbol{r}_{C}\times\boldsymbol{p}=\omega\left(\mathrm{I}_{C}\boldsymbol{\hat{z}}+mc^{2}\boldsymbol{\hat{z}}\right)
\end{aligned}$$
Consider the case where $\boldsymbol{\hat{z}}$ is one of the principal axes of rotation, with radius of gyration $R_{\rm gyr}^2$, then from above
$$ \boldsymbol{L}=\omega\,m\left(R_{{\rm gyr}}^{2}+c^{2}\right)\boldsymbol{\hat{z}} $$
and if you use this into (3) you will get
$$ \boxed{ \boldsymbol{r}_{X} = \boldsymbol{r}_{C}+\frac{R_{{\rm gyr}}^{2}}{c}\boldsymbol{\hat{x}}} \tag{4} $$
The axis of percussion is beyond the center of mass by a distance of $R_{\rm gyr}^2/c$.
A point particle has zero radius of gyration, so the percussion axis passes through the center of mass. The higher the radius of gyration is the further away from the center of mass the percussion axis is.
Also given a radius of gyration, the closer the center of mass is to pivot point, the further away the percussion axis is, and the other way around. The mimimum distance the axis of percussion is from the pivot point is $2 R_{\rm gyr}$ which occurs when the center of mass is one radius of gyration away from the pivot $c = R_{\rm gyr}$.