if they have less fitness they will be selected against.

I'm not disagreeing with you there. Being selected against is different than not being born in the first place.

Less fit directions have exactly as many direct children as more fit directions. The difference is that there are more grandchildren in more fit directions, and fewer in less fit directions. Depending on the parameters for how much variance there is from generation to generation, evolution can cross small (relative to mutations per generation) downward slopes in design space.

Let's have a ridiculously over-simplified example. There are four possible designs, AA, AB, BA, BB. AB and BA have value of .5, AA has value of 1, and BB has value of 2. Everyone starts at AA and has 1 child. 1% of children get moved from AA to AB or BA, from AB to either AA, or BB, from BA to either AA or BB. With 20k intial members, after one generation there's 200 members each of AB and BA. Those 200 members of AB have 100 children, one of which is BB. That lone member of BB has 2 children each generation until essentially everything is BB.

Ta-da, evolution managed to make it's way past a negative slope in design space and out of a local maxima.

I'm not saying that every local maxima can have some down-slope traversal, but that shallow enough moats can be crossed by random action. At this point, the now higher local maxima takes over.

If that's not enough, imagine that there are literally infinite starting members of a certain organism. After each generation, there will be an infinite number of each of every possible descendant. Even the less fit ones. It's trivial for evolution to get out of a local maxima with an infinite number of tries.