In order for us to see an object, our eye must receive light rays emitted from that object. For example, if I am standing in a dark room with no windows and no lights, you cannot see me because there are no light rays to bounce off of me towards your eye. Now if I light a candle, the light from that candle hits my face, and the light rays are deflected in all directions. Those light rays that travel towards you, enter your eye and allow you to see me. Thus to render something truly invisible to the human eye, one would need to intercept the light rays traveling from the object to the observer.
Some optical cloaking devices do this using lenses which deflect light around an object and making it harder for an observer to see. You can check those out in this
Now these devices are different from a true invisibility cloak in that, for now, they are directional. They only shield the object from view for an observer at a certain position. This is because the lenses aren't working in all directions in which the object emits light rays, only in certain directions. Physically, it is theoretically possible to create omnidirectional cloaking using lenses. It's just a matter of orienting the lenses such that they bend light rays for all angles of observation, and if this involves multiple different lenses, finding a way to make the transition between lenses look seamless.
An invisibility cloak would have to adapt its optics based on the position of the fabric at any given time. This might involve sensors that inform articulating lenses how to position themselves in order to deflect light properly. At present, adding sensors and motorized lenses would certainly add weight to an invisibility cloak and decrease its flexibility. In addition, being invisible to an unaided human eye, is not the same as being invisible to an infrared or ultraviolet camera. Because infrared and ultraviolet rays have different wavelengths than visible light, the lenses to deflect such rays are slightly different, but the problem of light rays is in principle the same. In essence, making an invisibility cloak is possible, it just comes down to a complex optical engineering challenge.