You know, I've thought about that many times too... In the end though, after taking some physics classes, I think everyone sees colors the same way, though don't quote me on this, I might be totally wrong. The colors we see are just different wavelengths of the visible spectrum (the entire spectrum ranges across radio waves to x-rays to gamma rays--visible light is only a very small fraction of that), ultra-violet being high-frequency (short wavelength) and infra-red being low-frequency (longer wavelength).

we all have rods and cones in our eyes, and when they are stimulated by a certain frequency I think each individual's brain should interpret them pretty much the same way, unless someone is color blind, of course.

Meh, I just wrote but I'm still not sure lol