More information on this can be found at here.

There have been two main approaches to colour associations with musical notes. Most stretch the spectrum of visual light across an octave, assigning adjacent colours of the spectrum to adjacent notes in the octave, using colour to code notes. Others, mot notably Scriabin's system associate harmonically close notes with adjacent colours of the spectrum, using colour to express something about harmonic relationship. Most of the examples that map the spectrum linearly across the octave are based on fairly subjective assignments of colours to notes. However, there is a plausible case for a particular relationship of this sort. Musical frequencies are conventionally organized by octave. Each octave represents a doubling of frequency. The frequencies of light wavelengths can be thought of as being many octaves above the range of human hearing. Also, the range of visible light occupies just a little less than a octave. The top end of the visible spectrum is a bit less than twice the frequency of the bottom end. By progressively dividing the frequencies of the visible spectrum by 2 until the frequencies enter the range of human hearing, a direct and not arbitrary mapping of colour to the notes of the octave is achieved. Whether there is any experiential significance to this particular mapping is unclear. A representation of this idea is presented here.

A very interesting presentation of the usefulness of Scriabin's "circle of fifths" approach is found in this video presentation by Stephen Malinowski, the creator the Music Animation Machine:

While various composers and musicians have proposed specific mapping of colour to note, there seems to be little coherence that would indicate an objective reason for a particular assignment. Colour, is a natural candidate for mapping some aspect of music. Colour also does have some experiential impact on its own. We do have different, and somewhat consistent physiological responses to colours. The attempts to use colour to express a musical composition's shifting moods or emotional content makes some sense in relation to this physiological response, however it is not clear that there is any inherent link between colour's expressiveness and any parameter of musical expression. On the other hand, it is clear that colour-coding the notes of the octave presents useful information to the viewer, especially where something like the circle of fifths is used to express something about tonality in the colour relationships.

Here is a field of slowly changing colour. Stare at it for a while to get a sense of what inherent expressive power colour on its own might have.