The reason the plunger pushes back is a difference in pressure between the interior of the plunger and the outside. Assuming we’re talking about an empty syringe and the end of the plunger is plugged, we are compressing some gas on the inside of the syringe. The only reason we can reduce the volume of the inside of the syringe is that we have applied a pressure on the plunger. As we push the plunger down though, the gas inside will resist that compression; that’s why we have to push down pretty hard on the plunger. Once we let go, the pressure on the inside of the syringe pushing up on the plunger (due to the compressed gas) is higher than the outside pressure from the outside environment. That imbalance in pressure (and therefore force, since Pressure = Force/Area) will cause the plunger to push right back up until the pressure inside is equal to the pressure outside.

This argument can be extended to the case where there is some fluid on the inside and can flow out of the needle. Fluids tend to be incompressible, meaning that as we push down with the plunger, it will be very hard to change the volume of the fluid. As a result, the fluid will flow out of the syringe. If we push down really hard and really fast though, the volume that we have displaced by pushing the plunger down is greater than the volume of the fluid that had the chance flow out of the syringe. In that case, once you let go, the plunger will also push back out once you let go because the fluid will apply pressure up on the plunger that is greater than the outside pressure.