This is a special post for quick takes by Oliver Sourbut. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
What constitutes cooperation?
Realised my model pipeline:
was missing an important preliminary step.
For cooperation to happen, you also need:
(Could break down further: identifying, getting common knowledge, and securing initial prospective cooperative intent.)
In some cases, 'identifying potential coalitions' might be a large, even dominant part of the challenge of cooperation, especially when effects are diffuse!
That applies to global commons and it applies when coordinating political action. What other cases?
'Identifying potential coalitions' is what a lot of activism is about, and it might also be a big part of what various cooperative memeplexes like tribes, religions, political parties etc are doing.
This feels to me like another important part of the picture that new tech could potentially amplify!
Could we newly empower large groups of humans to cooperate by recognising and fulfilling the requirements of this cooperation pipeline?
Here is some reasoned opinion about ML research automation.
Experimental compute and 'taste' seem very close to direct multiplier factors in the production of new insight:
My model of research taste is that it 'accumulates' (according to some sample efficiency) in a researcher and/or team by observation (direct or indirect) of experiments. It 'depreciates', like a capital stock, both because individuals and teams forget or lose touch, and (more relevant to fast-moving fields) because taste generalises only so far, and the 'frontier' of research keeps moving.
This makes experiments extremely important, both as a direct input to insight production and as fuel for accumulating research taste.
Peak human teams can't get much better research taste in absence of experimental compute without improving on taste accumulation, which is a kind of learning sample efficiency. You can't do that by just having more people: you have to get very sharp people and have very effective organisational structure for collective intelligence. Getting twice the taste is very difficult!
AI research assistants which substantially improved on experiment design, either by accumulating taste more efficiently or by (very expensive?) reasoning much more extensively about experiment design, could make the non-compute factor grow as well.
You can't just 'be smarter' or 'have better taste' because it'll depreciate away. Reasoning for experiment design has very (logarithmically?[2]) diminishing returns as far as I can tell, so I'd guess it's mostly about sample efficiency of taste accumulation.
(There's some parallelisation discount: k experiments in parallel is strictly worse than k in series, because you can't incorporate learnings.) ↩︎
A naive model where reasoning for experiment design means generating more proposals from an idea generator and attempting to select the best one has worse than logarithmic returns to running longer, for most sensible distributions of idea generation. Obviously reasoning isn't memoryless like that, because you can also build on, branch from, or refine earlier proposals, which might sometimes do better than coming up with new ones tabula rasa. ↩︎
Temporary MAP stance or subjective probability matching are my words for useful mental manoeuvres for research, especially when dealing with confusing or prepradigmatic or otherwise non-crisp domains.
MAP is Maximum A Posteriori i.e. your best guess after considering evidence. Probability matching is making actions/guesses proportional to your estimate of them being right (rather than picking the single MAP choice).
By this manoeuvre I'm gesturing at a kind of behaviour where you are quite unsure about what's best (e.g. 'should I work on interpretability or demystifying deception?') and rather than allowing that to result in analysis paralysis, you temporarily collapse some uncertainty and make some concrete assumptions to get moving in one or other direction. Hopefully in so doing you a) make a contribution and b) grow your skills and collect new evidence to make better decisions/contributions next time.
It happens to correspond somewhat to a decent heuristic called Thompson Sampling, which is optimal under some conditions for some uncertain-duration sequential decision problems.
HT Evan Hubinger for articulating his take on this in discussions about research, and I'm certain I've read others discussing similar principles on LW or EAF but I don't have references to hand.
If you want to be twice as profitable as your competitors, you don’t have to be twice as good as them. You just have to be slightly better.
I think AI development is mainly compute constrained (relevant for intelligence explosion dynamics).
There are some arguments against, based on the high spending of firms on researcher and engineer talent. The claim is that this supports one or both of a) large marginal returns to having more (good) researchers or b) steep power laws in researcher talent (implying large production multipliers from the best researchers).
Given that the workforces at labs remain not large, I think the spending naively supports (b) better.
But in fact I think there is another, even better explanation:
This also explains why it's been somewhat 'easy' (but capital intensive) for a few new competitors to pop into existence each year, and why firms' revealed preferred savings rate into compute capital is enormous (much greater than 100%!).
We see token prices drop incredibly sharply, which supports the non-commoditised margin claim (though this is also consistent with a Wright's Law effect from (runtime) algorithmic efficiency gains, which should definitely also be expected).
A lot of engineering effort is being put into product wrappers and polish, which supports the customer base claim.
The implications include: headroom above top human expert teams' AI research taste could be on the small side (I think this is right for many R&D domains, because a major input is experimental throughput). So both quantity and quality of (perhaps automated) researchers should have steeply diminishing returns in AI production rate. But might they nevertheless unlock a practical monopoly (or at least an increasingly expensive barrier to entry) on AI-derived profit, by keeping the (more monetisable) frontier out of reach of competitors?