My appeal is to our priors, or perhaps what they ought to have been (given some more logical coherence but not more evidence or hugely more logical coherence), before the past decade of observation. Then we should be able to agree that you made a large update, and I made a smaller update, and so I can legitimately ask you for the evidence that made you make the update.

It's plausible I believe AGI is much less difficult than you believe it to be, and thus I don't need as much evidence of current capabilities to believe we're close to it. How does one measure "difficulty" here? In the amount of "compute" required to achieve it, corrected for that compute's efficiency. Intelligent search by some of the smartest humans is incredibly efficient compared to evolution, and we now have evidence that scale works to make progress on almost any issue we care about.

The exception may be "creating useful and new concepts", but that can be explained by there being prerequisites which have only recently been met (notably, AI approaching autonomy at exploring idea space and evaluating which were useful). I'm very confident we're going to see useful and new concepts found by frontier AI within two years, and >50% within one year. The setup might involve a large loop which includes multiple training steps as it explores ideas.

Might you be interested in registering a nominal bet on the "useful and new" concept within one year?

By default you hit Amdahl's law

Qualitatively always true, but doesn't give us a quantitive analysis re AGI without looking into the details. Concretely, how important bottlenecks are depends on how reusable and general the existing pool of resources is. In a factory line with unique machines all calibrated to each other, increasing one 10x changes nothing. In a flexible factory floor with flexible humans and flexible tools, process improvements allow moving those humans around. I model the frontier AI companies as gaining a lot of available research power from their automation of the drudge work, that they can use to point at any problem including "have the AI be good at creating useful new concepts" and that it will fall once they do that.

[From here, refuting a Carl Shulman argument that we're increasing all inputs to intelligence]
On the conceptual AGI view, this isn't the case, because we aren't very greatly increasing the amount of great natural philosophers doing great natural philosophy about intelligence. That's a necessary input, and it's only being somewhat scaled up.

Evolution didn't need natural philosophers to build GI, so I don't understand the "necessary input" argument here. I believe efficient GI might require or greatly benefit from good natural philosophy, but that we'll be able to inefficiently do good natural philosophy with frontier AI within 3 years and thus in fact scale it up massively.

Glance at this section if you don't mind

I went ahead and read most of the debate, I'll respond to things there before responding to that section particularly.

First, some opening remarks:

• I hadn't read this previously, and now notice you've done more effort than I knew about to communicate your views, so my initial comment up-thread "they should make a more detailed case for this" was somewhat misinformed, sorry about that.
• I'm still unpersuaded by your lack of being persuaded by Demski and mine arguments, but am getting some impression that a bunch of the crux is about the idea that it's "high confidence" to believe 90%+ AGI within 5 years, and that you rely a lot on "forecasting is hard, thus we shouldn't update on vague evidence, thus we should remain on the vague prior"^[1] while I in fact believe ~3 year horizon forecasting for technological progress is doable (aka getting 90%+ that something will happen is something good superforecasters can do while being calibrated about it, even when the evidence looks weak to others who aren't trying as hard to guess the future), as evidenced by @Daniel Kokotajlo 's What 2026 looks like and currently resolved years for AI 2027.
  • Relatedly, it's not at all obvious that superforecasters need to be able to convince others of why they believe things, in order for them to be justfiably right. This could explain how reasonable people might see the same evidence and agree it's not convincing to you and yet still believe their confidence is warranted.
• Based on the above, and given your previous conversations not ending in agreement, it might be that our best resolution for this thread is eliciting near-term forecasts to let each other and the public assign Bayes Points soon.
• I had an agent research ur prev writings and predictions^[2] and I see u don't like making bets and predictions much. (It seems you have quite largely been right about those you've made, mostly in bio stuff). I generally have a similar vibe of disliking adversarial bets but do instead like sharing forecasts.

Now, reacting to some pieces, expliciting some more of where I'm coming from

1. PROBABLY, GI requires several algorithmic ideas. 
2. A priori, a given piece of silicon does not implement these algorithmic ideas. Like how piles of steel aren't randomly bridges.
2.1. So, a priori, PROBABLY GI is a rather long way off--hard to tell because humans given time can come with the algorithmic ideas, but they can't do much in a couple years. 
3. There's not a strong body of evidence that we have all / most of the algorithmic ideas. 
4. So we should have a fairly uninformed stance--though somewhat informed towards shorter--hard to say how much.
5. An uninformed stance implies smeared out, aka "long", timelines

1. Seems true for "efficient GI", but not generally true. Mutation and selection was enough to create some level of efficient GI, so itself looks like inefficient GI. Thus we already know some algorithms that can solve anything, if inefficiently. Thus we'll be able to implement inefficient GI without needing groundbreaking results, just cranking out efficiency improvements (as evolution did).
2. and 2.1 : Valid but unimportant if we already have the algorithms
3. True enough, it would take further argument to explain why current algos are good enough, and doing that in detail would be some form of capabilities research not much worth doing
4. This is valid. My position given my different arguments above is more of looking at quantitatively how many scaleups we should expect soon and then go "wow, that's a lot of scale ups!" and thus expect AGI soon by default.
5. Valid

In contrast, I am imagining that getting better at theorem-proving involves a lot of improvement to its chain-of-thought math tactics. At some point, to get better it will need to be inventing new abstractions in its chain-of-thought, on the fly, to improve its understanding of the specific problem at hand.

This paragraph gives valid requirements for LLM+harnesses being AGI alone, but doesn't constrain labs creating AGI using LLM+harnesses+more things very soon.

Consider how OpenAI found the Navier Stokes finite blowup and how it can do math training. A given LLM+harness doesn't need to do large innovations or invent very new abstractions in its chain-of-thought on the fly. As long as it produces valid incremental improvements that make it into training data, further derivations can be made. Likely the first AGI-like system will not be a given LLM&harness, but the whole continuous training setup they have in house to make progress on research questions.

In fact I think that (some?) mathematicians saying that there is increasing originality in the methods used for latest proof is evidence you would find valid, that recent developments in fact allow for some creativity of the sort you didn't see before.^[3] Of the many points made in this reply, this is one I'm most curious about your thoughts on.

Suppose there is a tweak to genetic code that will subtly tweak brain algorithms in a way which, when played out over 20 years (probably being expressed quadrillions of times), builds a mind that is better at promoting IGF. This tweak will be selected for. It doesn't matter if the tweak is about mental operations that take .5 seconds, 5 minutes, or 5 years. Evolution gets whole-organism feedback (roughly speaking). 

On the other hand, a single task execution in a training run for a contemporary AI system is very fast + short + not remotely as complex as a biotic organism's multi-year life. And that's where most of the optimization pressure is--a big pile of single task executions. 

Yes, this seems a correct description of things. It will be more difficult and take longer for frontier AI companies to make AI strong at long term tasks. If they only ever optimized for short term tasks, we would not magically see long term competence.

I believe they are well aware of this and are following an effective scaling up path where they first do easier things to get more money and resources to then attempt the harder things. It's furthermore my impression that they are being incredibly successful at this, as their recent models easily do advanced math within domain, and their frontier unreleased models can do some math somewhat outside domain (solving things no one knew how to solve despite trying).

If a frontier AI system takes years to become good at problems where humans also takes years, this won't mean we don't have AGI (it could be the problem is hard, or data constrained). If you think the best humans generally only invent one profoundly novel concept per 10 years, then can you rule out that we already have AGI but just haven't given it 10 years? How confident are you that tree and prune on the recent unreleased models doesn't yield new concepts if done sufficiently? If it is in fact the case that good concepts are mathematically good (eg. useful in more further stuff), won't they recognized when found, even if found randomly?

As an aside on the nature of GI, in your use of the word, in what sense are individual humans GIs? One architecture of the human brain allowed many different agents in varied and adapting environments to as a civilization recursively invent concepts and understanding and language and math etc, but this doesn't mean that a given human could do that even theoretically given extended lifetime.

If you agreed humans aren't that general, yet created all this, it'd be easier to see why frontier AI labs might soon have AGI overall, despite not cracking an algorithm that is AGI for a given instance. The more your model of GI includes environments and multi agent setups to provide the requisite data and incentives from which to learn the useful stuff and produce concepts in the first place, the more you can see we could build a lot of this in AI civilizations quite soon without needing any algorithmic breakthroughs. ^[4]

If you asked a deep thinker in 2015 what it would be like to talk to an entity that had solved a Millenium problem, hundreds of other notable open math questions, and had a professional level performance on tests of science, math, engineering, literature, history, etc. knowledge, they would be way way off about their expectations of what it would be like.

Sure, but if you ask Kokotajlo in 2022, he seems to get this much more right and be much less surprised. In fact I think several deep thinkers in 2022 correctly extrapolated from the progress that far to what we have now. Moravec's paradox has been known a while. Applying it quantitatively 10 years in advance is hard but 3 years in advance is quite doable, we have a bunch of data on what is or isn't hard for AI systems. Many people correctly and importantly validly predicted AI would be strong at Math proofs before it'd be strong at many other things, because of verifiable reward etc.

Overall, this is a long reply and I'm afraid you'd prefer not replying then attempt to reply to even half, so feel free to just reply to the most important or interesting parts.

1. ^{^}

   Of course, please clarify if I'm misunderstanding your position or that you think this is in fact not a crux

2. ^{^}

   I'm notably mentioning this wondering if you or people find this a rude thing to do.

3. ^{^}

   I have not looked into the proofs myself, and could see you coming up with counterarguments. Those in the genre of "it's not a new concept because it didn't give it a good name" are imo weak. If the proofs use (some) original methods that mathematicians go on to use in further applications, that should fit the novelty criteria. Identifying and naming the best concepts/methods an AI used in generating new results is something that could trained in later.

   If you disagree that there's increasing originality in the recent proofs, that's valid and I'm curious to know why.

4. ^{^}

   Aside to the aside, even stronger forms of intelligence may exist, I don't entirely rule out week-speed foom intelligence explosions from yet-unknown algorithmic findings (though I find them very unlikely). My argument aims to build a floor more than a ceiling, aka "humans aren't that general, there's no fundamental stuff missing to reach that level"