Temperature does in fact have an impact on a balloon. As long as the gas is ideal, the gas inside the balloon will follow this relationship:

(Pressure)*(Volume)=(Number of Molecules)*(a constant)*(Temperature).

This relationship is known as the ideal gas law. Now, let's say we increase the temperature of the balloon. In order to see the effects, we must hold all other variables constant (except the one we want to measure). In this case, let's keep the pressure constant (for a balloon, the number of molecules will always be constant since no gas molecules are leaving the balloon at any time.) If the pressure is constant and the temperature is increased, we see that to keep the equation true, the volume must also increase a proportional amount. On the other hand, if the volume was held constant (this would correspond to fitting the balloon in a container so it could not get bigger) then we would see the pressure against the walls of the balloon increase. Again, this would increase a proportional amount to how much temperature was increased.

If you look at the molecules within the balloon, our result makes sense. Adding temperature to the balloon would force the gas to have more energy. More energy would mean the molecules within the balloon would have to start moving faster. This increase in speed of the molecules would force the molecules to bounce off the walls of the balloon even harder. The result would then be an added pressure or volume to the balloon system.