This is a special post for quick takes by bpomo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Should we have a very exclusive AIS cause-prio sort of conference which outputs something like Hilbert Problems?
I still think something like “Hilbert problems for AIS” or just a convention of GOAT level researchers where they do super intense cause prio is quite good.
The idea:
- Get a number of very excellent AIS researchers/thinkers/funders (people with excellent end to end threat models/theories of change with various strategies) in the same place for 1+ days (IDK how long)
- Have people working on a very diverse set of things (e.g. China policy, specific control protocols, grassroots organizing, singular learning theory, etc. etc. etc. )
- Have them present their best arguments at the start of the thing for why more people should work on the set of things they think people should work on
- Have lots of 1 on 1s with goal of finding cruxes or changing minds
- Instruct people to make outcomes they want as specific/verifiably worded as possible. E.g. “scalable oversight” is not ok, but “models which we have confidence in as defined by X monitoring transcripts from all of the following operations inside of labs: Y% of internal sandboxed evals, Z% of blah blah” is better. This example is bad, but hopefully points in the direction of what I mean.
- If an outcome isn’t actually verifiable other researchers poke at it and try to help make it more verifiable. Maybe have some incentive structure IDK
- Maybe have some auction format where attendees put how much they’d pay, in percent of 2026-2028 cG dollars or some other valuation, to get various outcomes.
- Maybe have forecasters estimate the expected price of those outcomes in total labor/resources later and then you have a list of most cost effective things to try from this aggregate view
Pros:
- In general you get a list of very specific things people should work towards, and now instead of researchers saying “I work on scalable oversight!” they can say “I do this thing which I think will help achieve Guntherson’s criteria in the following way” and are more likely to be doing helpful stuff
- Important researchers at this conference might update their worldview to work on better things
- In general the conference runners record discussions and gather evidence the researchers reference as much as they can and can then present great steelman cases for various things to work on that people can look at
- Maybe this let’s you post cash prizes for some of the verifiable outcomes.
Cons:
- Hard to structure well?
- Cost + opportunity cost
- Maybe you make it less likely that researchers go exploring and find the good ideas that they haven’t thought of yet or otherwise limit future creativity. This seems not likely to be a huge cost to me: people who currently think outside of the box in terms of what we are prioritizing probably wouldn't stop because the list of things people currently care about is much better specified, ordered and presented.
1Thanks! I think this gets at: specification/verifiability is hard, but even thinking about it a little is useful.
- Main thing is either a snappy HTML site or Lesswrong post or all of the above with the final (verifiable! specific!) attendee supplied desired outcomes ranked by something like cost effectiveness or estimated value. If the event is good enough that notable attendees/cG or something/word of mouth signal boost it a lot, then this could be quite valuable.
- A deep dive into the list from 1., which looks like a longer version of 1. and includes some subset of:
- other options that didn't make the cut for the snappy list
- evidence and arguments that convinced attendees
- summaries of attendee reasoning/BOTECs which led to their specific valuations
- vibesy outcomes which were highly valued but hard to specify
- Improved relative valuations of various AI safety agendas in the minds of some attendees
- Maybe prize money attached to some of 1. by some funders, though this is tricky
I have low confidence here. I'd give ~30% that conditional on running something like the above, the most impactful output that the people running it are aware of is substantially distinct from any of the above. And I haven't thought much about how to optimize the above.
I know very little about it, but it seems like extremely little. From skimming the consensus "research priorities by area" list, it seems public facing and everything-bagel-y. They identify seven areas of research priority, and they are: cyber misuse, bio/chem, child safety, mental health/consumer protection, AI agents in the economy, open weight model safety and security, and (finally) loss of control and oversight.
Most of these things are not about reducing X risk at all, and I think it is clear that this group is thinking about different things than the people I'm envisioning, who are most worried about existential risk. Also, the leading proposals in the output are quite broad. I would want leading proposals from the output of the cause prio conference to be unusually specific.
The short answer is: sort of in vibes, not at all in practice, and the main issue is a waterline for participation that is too low in terms of seriousness about X-risk and openness/rationality.