Newton observed that color is not inherent in objects. Rather, the surface of an object reflects some colors and absorbs all the others. We perceive only the reflected colors.

Thus, red is not "in" an apple. The surface of the apple is reflecting the wavelengths we see as red and absorbing all the rest. An object appears white when it reflects all wavelengths and black when it absorbs them all.

We know we are seeing the same thing, because we have measured the different wavelengths of colors, and we all agree to call the wavelength of an apple, for example, red.