Wonderful question! As you rightly guessed, there is a close relationship between the color and brightness of light and how it heats up surfaces that it hits. Assuming that the light your mirror is reflecting comes from the sun, we can describe the 'color' of sunlight in terms of its components: visible light (light your eyes can see), ultraviolet light (invisible light that gives you sun burns) and infrared light (invisible and harmless, but can make you feel warm). When you sit outside in the sun, most of the sunlight (UV, visible and infrared) is absorbed by your clothes, skin and body. When light is absorbed it generally turns into heat, and this is true for any color of light. So, the heat that you feel from sitting outside actually comes from all colors of sunlight (UV, visible and infrared) being absorbed and converted into heat.

Back to your question: if your towel is absorbing light before it is reflecting off the mirror, then it too will heat up, and block the reflected light from heating up other surfaces. The color of the towel (blue-green, for example) is a sign of how much visible sunlight your towel is absorbing or reflecting back to your eyes. A completely black towel would absorb all of the visible light, while a colorful blue-green towel absorbs red-yellow colors more than blue-green. So, we can expect that the darker towel would be slightly better at absorbing sunlight than the blue-green towel. Both the darker and colorful towels will absorb most of the visible light however, and you probably won't be able to notice that much of a difference.

To illustrate this point, compare a towel to a window or a mirror: any color of towel will block visible light from passing through, and only the brightest towels will reflect enough sunlight to make you squint when you hold them up against the sun.

However, this description completely ignores the other colors of sunlight that our eyes cannot see: UV and infrared light. Visible light only accounts for about half of the energy (heating power) of sunlight. 10% of the energy is in the UV, and 40% of the energy is infrared light. That means infrared light can create a lot of heat when it is absorbed by a material, but we won't be able to tell with our eyes. Your towel may or may not be able to absorb a lot of infrared light. The infrared 'color' of your towel depends on the material that it's made of. Cotton tends to absorb more infrared light than polyester, but this depends on how tightly the towel is woven, how thick it is, and how much water it has absorbed.

You can test how much infrared light your towel absorbs by standing in its shadow during a sunny day. If you can still feel the sunlight warming you up, then most of the infrared light is passing through. However, if the towel keeps you cooler then it is more effective at blocking infrared light. This same logic is used to design windows in modern buildings. Most new windows are coated with a very thin layer of silver, or another reflecting metal, that can block infrared light without blocking too much visible light. This lets you stand next to the window without feeling as hot from the infrared sunlight, and keeps the room much cooler. The UV portion of the sunlight can also heat up a material, but most objects that are colorful to the eye are good at absorbing UV light. Also, glass is good at blocking UV light, so it is unlikely that you will ever get a sunburn behind a window. Finally, if you are trying to keep your room cool, keep in mind that the towel on the mirror will still get warm, and eventually heat up the room, when it absorbs sunlight. The best approach to keep the room cool is to use your blinds to block or reflect the sunlight before it ever enters the room.

Covering the mirror with a towel will prevent the mirror from receiving much light and therefore prevent the mirror from reflecting much light. However, I wonder if the heat you feel from the mirror is because of the heat from a lamp near the mirror in the first place. If that is the case, covering the mirror with something that absorbs light and heat will help not redirect the heat, but not solve the fundamental problem, which is the lamp (or any other heat source that can be reflected by the mirror). Finally, I'm not sure that the mirror can be called "harmful". It may contribute to the temperature of the room, but heat itself is not harmful to us unless it is excessive.

Mirros are not harmful. They reflect light, which carries energy, and therefore heat, but they are not dangerous.