I'll put this in another way, perhaps it will help someone to finally grasp it.
THERE ARE NO COLORS
There are just rays of light of different frequencies. And there's an infinite variety of frequencies available.
Where colors appear are in our eyes and in our minds. In our eyes we have the photoreceptor cells, and our minds interpret the signals from these cells. Furthermore, you probably already know that there are 4 types of these cells in our eyes - one senses just brightness, the other three sense ROUGHLY red, green and blue wavelengths. Although it's really more complicated. Check this out:

As you can see, each cell doesn't just respond to a single frequency, but rather to a large range of frequencies, and these ranges overlap A LOT.
In the end, the color that we perceive is determined by the RATIO of the stimulation of all three types of cells. You'll also note that some colors like white or brown aren't even on this chart, because they require more than one frequency of light rays at the same time.
Other animals can have more or less than 3 types of cells, and the colors they perceive will be drastically different from ours. Check out the mantis shrimp which has 12 to 16 types of color receptor cells!!!
Interestingly, there are even some humans with 4 types of cells, although a fully functioning tetrachromacy is very rare.
So back to RGB and CMYK and HSL and whatnotelse. These notations are used because they are convenient ways to describe most colors we humans can perceive. Enough for practical purposes*. But none of them actually cover ALL of the colors that we can perceive. Similarly, I also don't think we've created a device that could create all the possible colors that humans can perceive (but I could be wrong on this one).
And if we wanted to create a mathematical representation of all the colors that a mantis shrimp can see, we'd need 16 components instead of 3 components.
As for treating colors as 3D vectors - uhh, sure? You can do that, mathematically. What that will give you however, I don't know.
*Added: as noted in comments by Edgar Bonnet, there is the CIE 1931 XYZ color space which does cover all possible colors that humans can see.
float
s, which have even smaller (and non-even) spacing. – Ruslan Apr 01 '21 at 23:14