No one knows for sure. One idea is that VERY EARLY in its history, Venus was clobbered by a planet , about 20 % the present-day mass of Venus, and that this collision resulted in the retrograde rotation of present day Venus. The spin of a planet endows a planet with something called angular momentum. A spinning basketball for example has a certain amount of angular momentum that depends on the spin rate and the mass of the ball. The angular momentum is not a scalar quantity however. That is , the angular momentum is a quantity that has a magnitude AS WELL AS DIRECTION. So that a spinning basketball has an angular momentum vector that points up. Now if this basketball is hit with a soccer ball coming from a direction opposite of the spin direction, when the angular momentum of the basketball gets modified by the angular momentum of the soccer ball due to collision, the final state of spin of the basketball can be very different in magnitude AND direction compared to what it was BEFORE the collision took place.

Another factor is that because Venus is so close to the SUN, the TIDES raised on Venus by the Sun could have braked its initially higher rotation rate...but tides could not turn the angular momentum vector around. At any rate, this is a very good question!