The question is very well worded because it acknowledges that there are individual differences in this phenomenology of associating a color with an emotion. A related psychological phenomenon is synesthesia and it refers to a certain sensory experience giving rise to an experience from another sense. For example, individuals that experience this phenomenon, synesthetes as they are called, report seeing a color and being able to taste it. In that case, a color (visual) sensory experience is associated with an olfactory (smell) or gustatory (taste) experience. There are other forms of sensory substitution.

It is not clear why this happens. Individuals that experience this might have different brain networks. For example, the person that experiences a taste once she sees a color might have strong connections between V1 in the occipital cortex (vision for color) and the olfactory bulb (smell and taste).

Although it is plausible that the reason for this is due to the brain, other explanations should also be entertained. For example, emotion is a higher level more complex psychological phenomena compared to taste. It is possible that certain cultures have different emotional connections with colors, and it is not clear whether they have different brains. The point is that the neurobiological explanation should not be the most convincing just because it is more 'scientific.' We can provide different explanations for different levels.

Your question is one that scientists are asking.We're just starting to be able to get interesting images of our brains while we're thinking or doing things. Some day we'll be able to see what areas of the brain are active when we look at a specific color.

Meanwhile you can ask your friends and other people to look at colors and tell you what they think. The answer might be different for different people. I like red; but I had a lesson about technical writing from a woman, and she said never to use a red pen, because that was an angry color.