I am not adding more detail to my prediction, I'm adding more detail to my justification of that prediction, which doesn't make my prediction less probable. Unless you think predictions formed on the basis of little information are somehow more robust than predictions formed based on lots of information.

As for denying the super-exponential trend, I agree. I don't put a lot of stock in extrapolating from past progress at all, because breakthroughs are discontinuous. That's why I think it's valuable to actually discuss the nature of the problem, rather than treating the problem as a black box we can predict by extrapolation.

"having another person reflect your situation back to you" sounds exactly like "paid friend"

I do suspect the reason everyone needs therapists now is that we've destroyed our communities in the west and are in the middle of a loneliness epidemic. Though a therapist is a friend who spends all day talking to people about their issues, so they're probably particularly good at it.

I think therapy is probably around as helpful as exercise, for example, but you might be foolish not to do both if the effect size of both is significant enough to make both worthwhile. They're independent, and doing one doesn't rule out the other. Having a therapist to keep you accountable also helps you stick to things long-term.

Anyway, the cost of trying it is very low compared to the possible payoff.

I went into more detail about why I think this is more than 10 years away in a follow-up blog post:

https://www.lesswrong.com/posts/F7Cdzn5mLrJvKkq3L/shallow-vs-deep-thinking-why-llms-fall-short

It's funny everyone is doubting the funny jokes part. I view funny jokes as computationally hard to generate, probably because I've sat down and actually tried, and it doesn't seem fundamentally easier than coming up with brilliant essay ideas or whatever. But most people just have experience telling jokes in the moment, which is a different kind of non-deep activity. Maybe AI will be better at that, but not so good at e.g. writing an hour of stand-up comedy material that's truly brilliant?

For example, for things like "LLMs are broadly acknowledged to be plateauing", it's probably going to be concurrently both true and false in a way that's hard to resolve

Yes, this is somewhat ambiguous I admit. I'm kind of fine with that though. I'm not placing any bets, I'm just trying to record what I think is going to happen, and the uncertainty in the wording reflects my own uncertainty of what I think is going to happen.

I think that in theory there is nothing wrong with having your memory wiped every iteration, and that such an architecture could in theory get us to SC. I just think it's not very efficient and there would be a lot of repeated computation happening between predicting each word.

I'm not confident neuralese is more than a decade away. That could happen by 2027 and I wouldn't be shocked. I don't think it'll be a magic bullet though. I expect less of an impedance mismatch between neuralese and the model than language and the model, but reducing that impedance mismatch is the only problem being solved by neuralese.

To be clear, I think that basically any architecture is technically sufficient if you scale it up enough. Take ChatGPT, make it enormous, through oceans of data at it, and then allow it to store gigabytes of linguistic information. This is eventually a recipe for superintelligent AI if you scale it up enough. My intuition so far is that we basically haven't made any progress when it comes to deep thinking though, and as soon as LLMs start to deviate from learned patterns/heuristics, they hit a wall and become as smart as a ~shrimp. So I estimate the scale required to actually get anywhere with the current architecture is just too high.

I think new architectures are needed. I don't think it will be as straightforward as "just use recurrence/neuralese", though moving away from the limitations of LLMs will be a necessary step. I think I'm going to write a follow-up blog post clarifying some of the limitations of the current architecture and why I think the problem is really hard, not just a straightforward matter of scaling up. I think it'll look something like:

Each deep problem is its own exponential space, and exploring exponential spaces is very computationally expensive. We don't do that when running LLMs for a single-pass. We barely do it when running with chain of thought or whatever. We only do it when training, and training is computationally very expensive, because exploring exponential spaces is very computationally expensive. We should expect an AI that can generically solve deep problems will be very computationally expensive to run, let alone train. There isn't a cheap, general-purpose strategy for solving exponential problems, so you can't re-use progress from one to help with another necessarily. An AI that solves a new exponential problem will have to do the same kind of deep thinking AlphaGo Zero did in training when it played many games against itself, learning patterns and heuristics in the process. And that was a best-case, because you can simulate games of Go, but most problems we want to solve are not simulate-able, so you have to explore exponential space in a much slower, more expensive way.

And btw I think LLMs currently mostly leverage existing insights/heuristics present in the training data. I don't think it's bringing much insight on it's own right now, even during training. But that's just my gut feel.

I think we can eventually make the breakthroughs necessary and get to the scale necessary for this to work, but I don't see it happening in five years or whatever.

Yeah, someone pointed out that footnote to me, and I laughed a bit. It's very small and easy to miss. I don't think you guys actually misrepresented anything. It's clear from reading the research section what your actual timelines are and so on. I'm just pointing to communication issues.

Thanks for your responses! I'll check them out.

AI might have transformed some major industries and careers, even without providing novel research or human level insights.

I definitely agree this will happen a lot sooner than superhuman coders. Growth of shallow reasoning ability is not enough to cure cancer and bring about the singularity, but some things will get weird. Some careers will probably vanish outright. I often describe some careers as "requiring general intelligence", and by that I mean requiring deep thinking. For example, writing fiction or doing AI research. In a sense, when any one of these falls to AI, they all fall. Until then, it'll only be the shallow jobs (transcription, for example) that can fall.

I should also note that while this puts my odds of 2027 Foom and Doom in the single digits or lower... that's still an awful high figure for the end of all humanity. Flip a coin 7-9 times. If it's head each time, then every one of us will be dead in 3 years.

Agreed

I'm not really talking about true information loss, more like the computation getting repeated that doesn't need to be.

And yes the feedforward blocks can be like 1 or 2 layers deep, so I am open to this being either a small or a big issue, depending on the exact architecture.