Have you seen the classic parody article “On the Impossibility of Supersized Machines”?

No, lol. That's a good one.

The latter.

Well, if we extrapolate from the current progress, soon AI will be superhumanly good at complex analysis and group theory while only being moderately good at ordering pizza.

That's why I think that comparing AI to humans on a one-dimensional scale doesn't work well.

To be clear, I'm not claiming that the "write a text with precisely X words" task is super-duper-mega-hard, and I wouldn't be surprised if a new frontier model was much better at it than Gemini. I have a very similar opinion to the author of this post: I'm saying that given what the models currently can do, it's surprising that they also currently can't (reliably) do a lot of things. I'm saying that there are very sharp edges in models' capabilities, much sharper than I expected. And the existence of very sharp edges makes it very difficult to compare AI to humans on a one-dimensional intelligence scale, because instead of "AI is 10 times worse than humans at everything", it's "AI is roughly as good as expert humans at X and useless at Y". 

There's lots of coding training data and not very much training data for creating documents of a specific length. I think if we added a bunch of "Write ### words about X" training data the LLM's would suddenly be good at it.

My point was that it's surprising that AI is so bad at generalizing to tasks that it hasn't been trained on. I would've predicted that generalization would be much better (I also added a link to a post with more examples). This is also why I think creating AGI will be very hard, unless there will be a massive paradigm shift (some new NN architecture or a new way to train NNs).

EDIT: It's not "Gemini can't count how many words it has in its output" that surprises me, it's "Gemini can't count how many words it has in its output, given that it can code in Python and in a dozen other languages and can also do calculus".

they put substantial probability on the trend being superexponential

I think that's too speculative.

I also think that around 25-50% of the questions are impossible or mislabeled. 

I wouldn't be surprised if 3-5% of questions were mislabeled or impossible to answer, but 25-50%? You're basically saying that HLE is worthless. I'm curious why. I mean, I don't know much about the people who had to sift through all of the submissions, but I'd be surprised if they failed that badly. Plus, there was a "bug bounty" aimed at improving the quality of the dataset.

TBC, my median to superhuman coder is more like 2031.

Guess I'm a pessimist then, mine is more like 2034.

I don't have one knock-down counterargument why the timelines will be much longer, so here's a whole lot of convincing-but-not-super-convincing counterarguments:

1. This contradicts METR timelines, which, IMO, is the best piece of info we currently have for predicting when AGI will arrive.
2. Microsoft is not going to fund Stargate. At best, it means Stargate will be delayed. At worst, it means it will be axed. For the timelines in this post to be accurate, Stargate would have to be mostly finished right now, today. Even if OpenAI could literally print money, large data centers take 2-6 years to build.
3. "Moreover, it [Agent-1] could offer substantial help to terrorists designing bioweapons, thanks to its PhD-level knowledge of every field and ability to browse the web."
   "Agent-1, a new model under internal development, it’s good at many things but great at helping with AI research."
   Considering that frontier LLMs of today can solve at most 20% of problems on Humanity's Last Exam, both of these predictions appear overly optimistic to me. And HLE isn't even about autonomous research, it's about "closed-ended, verifiable questions". Even if some LLM scored >90% on HLE by late 2025 (I bet this won't happen), that wouldn't automatically imply that it's good at open-ended problems with no known answer. Present-day LLMs have so little agency that it's not even worth talking about.
   EDIT: for the record, I expect 45-50% accuracy on HLE by the end of 2025, 70-75% by the end of 2026, and 85-90% accuracy by the end of 2027.
4. A memory module that can be stored externally (on a hard drive) is handwaved as something that Just Works™, I don't expect it to be so easy.
5. As of today, there is no robust anti-hallucination/error correction mechanism for LLMs. It seems like another thing that is handwaved as something that Just Works™: just beat the neural net with the RLHF stick until the outputs look about right.
6. Imagine writing a similar piece for videogames in 2016, after AlphaGo became news. If someone wrote that by 2018-2019 all mainstream videogames would use neural nets trained using RL, they would be laughably wrong. Hell, if someone wrote that but replaced 2018-2019 with 2024-2025, they would still be wrong.
   This is the least convincing argument of all of these, it's just my way of saying, "I don't feel like reality actually, really works this way". On that nore, I also expect that recursive self-improvement requires a completely new architecture that not only doesn't look like Transformers but doesn't even look like a neural network.

Not a criticism, but I think you overlooked a very interesting possibility: developing a near-perfect speech-to-text transcription AI and transcribing the entire YouTube. The biggest issue with training multi-modal models is acquiring the right ("paired") training data. If YouTube had 99.9% accurate subtitles for every video, this would no longer be a problem.

You might be interested in reading about aspiration adaptation theory: https://www.sciencedirect.com/science/article/abs/pii/S0022249697912050

To me the most appealing part of it is that goals are incomparable and multiple goals can be pursued at the same time without the need for a function that aggregates them and assigns a single value to a combination of goals.

I'm quite late (the post was made 4 years ago), and I'm also new to LessWrong, so it's entirely possible that other, more experienced members, will find flaws in my argument.

That being said, I have a very simple, short and straightforward explanation of why rationalists aren't winning.

Domain-specific knowledge is king.

That's it.

If you are a programmer and your code keeps throwing errors at you, then no matter how many logical fallacies and cognitive biases you can identify and name, posting your code on stackoverflow is going to provide orders of magnitude more benefit.

If you are an entrepreneur and you're trying to start your new business, then no matter how many hours you spend assessing your priors and calibrating your beliefs, it's not going to help you nearly as much as being able to tell a good manager apart from a bad manager.

I'm not saying that learning rationality isn't going to help at all, rather I'm saying that the impact of learning rationality on your chances of success will be many times smaller than the impact of learning domain-specific knowledge.

Ok, thank you for the clarification!