An exponent models things locally, at an appropriate level of detail for modeling them locally. An S-curve won't actually be an S-curve, there will be a lot more data than that in the real thing, omitting the data specifying when the exponent slows down is no different. Simpler models are often more useful, even when you do realize they have a limited scope of applicability.

This is absolutely true. However, actually using it effectively requires having a sufficiently good, principled reason for thinking the limit is in some particular place, or will be approached on some particular timeline. When I've looked at many (most?) real-world attempts to forecast when some particular exponential will start looking like an s-curve, they're usually really far off, sometimes in ways that 'should' be obvious, even if the forecasting exercise itself is instructive.

Low effort, low nuance hot take:

Sigmoids are fake. Real curves aren't symmetrical, and research progress curves in particular tend to look like exponentials that run into brick walls. Adoption curves can be relatively balanced but I still don't buy it.

And given you already can't fit a sigmoid that isn't already dying and probably shouldn't try, fitting the thing that actually happens in reality is doomed. Fit your exponential and intuit the rest. You don't actually have quantified evidence for when it ends, so use the evidence you do have instead.

I think you’re just not appreciating the usefulness of partial world models. 
A sigmoid often looks locally exponential. If the local part is all you can model effectively, your model looks exponential. But it’s a local model. It’s not trying to make claims about the part where the sigmoid flattens out. Why include that part in your model when you don’t have a basis for drawing conclusions about it? It will rather tend to make you more wrong than less wrong :)

Whilst technically true, those who attempt to use this fact to predict future growth tend to end up just as wrong as those who don't.

All exponentials end, but if you don't know when you need to prepare equally for the scenario where it ends tomorrow, and the one where it ends with the universe in paperclips.

Yes, if you want to model growth, you probably want something like a sigmoid or Gompertz curve, whose value approaches some limit rather than rising infinitely forever.

But if you want to actually draw that curve, and use it to predict effects that you care about, you need to know what limit it approaches.

Whether that curve approaches 1 or 10^100, is going to matter a lot.

You say "if we are to accurately model the world"...

If I am modelling the path of a baseball, and I write "F = mg", would you "correct" me that it's actually inverse square, that the Earth's gravitation cannot stay at this strength to arbitrary heights? If you did, I would remind you that we are talking about a baseball game, and not shooting it into orbit—or conclude that you had an agenda other than determining where the ball lands.

What if I'm sampling from a population, and you catch me multiplying probabilities together, as if my draws are independent, as if the population is infinite? Yes there is an end to the population, but as long as it's far away, the dependence induced by sampling without replacement is negligible.

Well, that's the question, whether to include an effect in the model or whether it's negligible. An effect like finite population size, diminishing gravity, or the "crowding" effects that turn an exponential growth model logistic.

And the question cannot be escaped just by noting the effect is important eventually.

Communication is about other people. It is the thoughts that we put into other's heads, not the precise content of the propositions themselves. The graphs and other symbols we choose to use should put more likely-correct and contextually relevant thoughts into other people's heads.

If I am communicating with someone because I believe their lack of information gives them an insufficiently aggressive model of past (and therefore probably future) AI capabilities progress, then the only reason I can think to use the sigmoid graph is to head off this specific criticism so that they are not distracted by the actual content of the communication. If I did make this choice, I would also want to put a cut in the y axis and cheekily label the upper value with something to indicate that it may be far beyond what they are currently entertaining to be possible.

I often do make this choice when verbally communicating. e.g. "Of course, it has to level off at some point, but that could very well be several orders of magnitude above human capability." If I have exactly one image and no chance to elaborate, for most audiences, a sigmoid probably does a poorer job of putting correct and relevant thoughts into their heads than an exponential.

But which S curve? People like exponentials because of the differential equation they satisfy. To choose a type of S curve you're implicitly choosing a differential equation and you have justify that choice.

I agree that all exponentials eventually end, but I don't think this matters that much, because trying to predict the future using sigmoids/functions that don't grow without bound is also very hard, and you need to have very good priors or very low noise data to not be very off, so an exponential is usually a better fit unless you have reason to believe you've identified a specific upper bound and the dampening term:

Why sigmoids are so hard to predict

Sigmoids behaving badly: why they usually cannot predict the future as well as they seem to promise (paper)

A key crux is I basically disagree with the assumption that you can reliably detect either the dampening term or the upper bound in early data, because I expect real-life data to be far too noisy to figure both of these out, and this is especially the case in AI. 

I see this comment as making the assumption that you can actually just guess an upper bound without being OOMs wrong in the general case.

There are special cases like disease modeling (though even there predictions can be wildly off), but in general I think the exponential fit is almost always better than a sigmoid/function that doesn't grow without bound unless you have strong reason to believe you've identified both the dampening term and the upper bound.

Some curves that start exponential are actual unimodal, in the long run. Some are oscillatory. Some are chaotic, some have undefined behaviour because the thing being plotted ceases to make sense.

Why do you think all of them are eventually S curves in particular?