While a balloon may seem like it's impenetrable, if look at it with a microscope you would see that it is in fact very porous. This means that there are lots of very tiny holes in the balloon. In addition, Helium atoms are very, very small - around 31 Pico meters. While the balloon seems like it's holding the gas for a long time, it actually begins to leak as soon as you fill it up. The holes in the balloon allow the gas to slowly leak out and escape.

To better visualize this, you can think of a large bucket full of water. If there are a few very small holes in the bottom of the bucket, the water will slowly leak out. You may not see the level changing very quickly but if you come back in a few hours you will definitely see a change just as you will see a change in the helium balloon a day or even a few hours later.