You are basically correct. For a ball to roll from one point to a lower point that is not directly below it following a track that is say mounted on a flat wall, it gets to the lower point faster if the track is not a straight line and instead is initially more in the downward direction than in the direction of the endpoint. This gives the ball acceleration right away so that it has higher average velocity.

Now you may want to know what track shape is the best and I believe it is a cartioid curve if my memory serves me correctly (this curve gets its name from its resemblance to a heart). Interestingly, if I recall my history correctly, it was Isaac Newton who first solved this problem. In fact there was a contest in Europe and supposedly many great minds were trying to answer this question. I am not certain if the story is correct, but what I read was that someone visiting Newton in England told him about the contest while having dinner. Newton found the correct answer before dinner was over.