Nice! I'm very late to this. A few thoughts.

Focusing on 'full agents' might be misleading. Humans are fairly agenty as far as things go. Mobs and corporations and countries are a bit less agenty on the whole. So maybe you need some spectrum (or space) of agentness for things to be appropriately similarly-typed.

Game theory and MDPs and so on often treat the population as static. But we spawn (and dissolve) actors all the time. So a full theory here would need to build that in from the start. This has the nice property that it might be a foundation to describe a hierarchy too: mobs, corporations, and countries get spawned, dissolved, transformed, etc. all the time, after all. (Looser thought: could some agents-composed-of-parts be 'suspended' or 'transferred' to different substrate...?)

(Maybe you're looking for the word 'seasoning'...? But maybe that includes other herbs in a way you didn't want.)

With that weighting, human experts have much less frontier-taste-relevant data than they have total data... AI can learn from all copies.

Ah, yep yep. (Though humans do in fact learn (with losses) from other copies, and within a given firm/lab/community the transfer is probably quite high.)

Hmm, I think we have basically the same model, with maybe a bit different parameters (about which we're both somewhat uncertain). But think that readers of the OP without some of that common model might be misled.

From a lot of convos and writing, I infer that many people conflate a lot of aspects of intelligence into one thing. With that view, 'intelligence explosion' is just a dynamic where a single variable, 'the intelligence' (or 'the algorithmic quality'), gets real big real fast. And then of course, because intelligence is the thing that gets you new technology, you can get all the new technology.

About this, you said,

There is then a further question, if that IE goes very far, of whether AI will generalize to chemistry etc

I'd have thought "yes" bc you achieve somewhat superhuman sample efficiency and can quickly get as much chemistry exp as a human expert

revealing that you correctly distinguish different factors of intelligence like 'sample efficiency' and 'chemistry knowledge' (I think I already knew this but it's good to have local confirmation), and that you don't think a software-only IE yields all of them.

Regarding the second sentence, it could be a misleading use of terms to call that 'generalisation'^[1], but I agree that 'sample efficiency' is among the relevant aspects of intelligence (and is a candidate for one that could be mostly generalisably built up automated in silico), and a relevant complement is 'chemistry (frontier research) experience', and that a lot of each taken together may effectively get you chemistry research taste in addition (which can yield new chemistry knowledge if applied with suitable experimental apparatus).

I'm emphasising^[2] (in my exploration post, in this thread, and the sister comment about frontier taste depreciation) that there's practically a wide gulf between 'hoover taste up from web data' and 'robotics or humans-with-headsets', in two ways. The first tops out somewhere (probably at sub-frontier) due to the depreciation of frontier research taste. The second cluster doesn't top out anywhere, but is slower and has more barriers to getting started. Is a year to exceed humanity's peak taste bold in most domains? Not sure! If a lot of in silico is possible, maybe it's doable. That might include cyber and software (and maybe narrow particular areas of chem/bio where simulation is especially good).

If you know for sure that those other bottlenecks proceed super fast, you don't need to necessarily clarify what intelligence factors you're talking about for practical purposes, but if you're not sure, I think it's worth being super clear about it where possible.

Yes more 'data', but not necessarily more frontier-taste-relevant experience, which requires either direct or indirect contact with frontier experiments. That might be a crux.

BTW I also think novelty taste depreciates quickly as the frontier of a domain moves, so I'm less bullish on hoovering up taste from existing data/interviews for a permanent taste boost. But it might take you some way past frontier, which may or may not be sizeably impactful in a given domain.

I think we're on the same page about what factors exist.

For the intelligence explosion I think all we need is experience with AI RnD?

Well, sort of, but precisely because intelligence isn't single-dimensional, we have to ask 'what is exploding here?'. And like I said, I think you straightforwardly get speed. And plausibly you get sample efficiency (in the inference-from-given-data sense). And maybe you get 'exploratory planning', which is hypothetically one, more transferrable, factor of general exploration efficiency.

But you don't get domain-specific novelty/interestingness taste, which is another critical factor, except for the domains you can easily get lots of data on. So either you need to hoover that up from existing data, or get it some other how. That might be interviews, it might be a bunch of robotics and other actuator/sensor equipment, it might be the Shulman-style humans-with-headsets-as-actuators thing and learning by experience. But then the question becomes whether there's enough data to hoover it up real fast with your speed+efficiency learning, or whether you hit a bunch of scaling up bottlenecks (and in which domains).

I am hopeful that one of the things we can do with just-before-the-brink AI will be to accelerate the design and deployment of such voluntary coordination contracts. Could we manage to use AI to speed-run the invention and deployment of such subsidiarity governance systems? I think the biggest challenge to this is how fast it would need to move in order to take effect in time. For a system that needs extremely broad buy-in from a large number of heterogenous actors, speed of implementation and adoption is a key weak point.

FYI FLF has a Fellowship on AI for Human Reasoning which centrally targets objectives like this (if I've understood)

I wrote a bit about experimentation recently.

You seem very close to taking this position seriously when you talk about frontier experiments and experiments in general, but I think you also need to notice that experiments come in more dimensions than that. Like, you don't learn how to be better at chemistry just by playing around with GPUs.

It's quite clear that labs both want more high-quality researchers -- top talent has very high salaries, reflecting large marginal value-add.

Three objections, one obvious. I'll state them strongly, a bit devil's advocate; not sure where I actually land on these things.

Obvious: salaries aren't that high.

Also, I model a large part of the value to companies of legible credentialed talent being the marketing value to VCs and investors, who (even if lab leadership can) can't tell talent apart except by (rare) legible signs. This is actually a way to get more compute (and other capital). (The legible signs are rare because compute is a bottleneck! So a Matthew effect pertains.)

Finally, the utility of labs is very convex in the production of AI: the actual profit comes from time spent selling a non-commoditised frontier offering at large margin. So small AI production speed gains translate into large profit gains.

The best objection to an SIE

I think the compute bottleneck is a reasonable objection, but there is also the fairly straightforward objection that gaining skills takes experience, and experience takes interaction (or hoovering up from web data).

You can get experience in things like 'writing fast code' easily, so a speed explosion is fairly plausible (up to diminishing returns). But various R&Ds or human influence or whatever are much harder to get experience for. So our exploded SIs might be super fast and maybe super good at learning from experience, but out of the gate at best human expert level where relevant data are abundant, and at best human novice level where data aren't.