When an object appears a certain color when illuminated by white light it means that it is reflecting light of that color and absorbing all other colors. For example, a red apple is reflecting red light and absorbing all other colors of light. The more light the object absorbs, the more heat absorbed since light is energy. If you consider it a color, black absorbs the most heat. A black object absorbs all wavelengths of light and reflects none. Objects that are white, on the other hand, reflect all wavelengths of light and therefore absorb the least heat.

Now, what about a color like yellow? It is difficult to compare the amount of heat absorbed by a yellow object verses another colored object like green or red because a lot of other factors come into play. For example, the intensity of the color, the incident illumination (the wavelengths of light hitting the object) and the angles of the light coming in and being reflected all play a part in the amount of heat absorbed. Now if you have a light yellow T-shirt and a dark yellow T-shirt, you can confidently say that the darker (more intense) yellow will absorb more heat because there is more pigment (less white). But comparing to another colored T-shirt like blue-green or red you would have to experimentally test!