Such as that, on the fairly plausible assumption that the human race will eventually colonize the stars, as long as we manage not to go extinct first, it seems rather likely that that will be something of at least the rather rough order $O\left({10}^{\sim 10}\right)$ times as many people in our current forward lightcone as in our backward lightcone. That’s a rather large coincidence — why is our current viewpoint that atypical?

I think the most likely solution is that your current viewpoint is actually typical. Most human experience in the universe is being a key figures in the transition to terrestrial superintelligence.

I'm not seeing how you are drawing that from my numbers

People sometimes talk about "alignment by default" — the idea that we might solve alignment without any special effort beyond what we'd ordinarily do. I think it's useful to decompose this into three theses, sorted from strong to weak:

1. Alignment by Default Techniques. Ordinary techniques for training and deploying AIs — e.g. labelling data to the best of their ability, using whatever tools are available (including earlier LLMs) — are sufficient to produce aligned AI. No special techniques are required.
2. Alignment by Default Market. Maybe default techniques aren't enough, but ordinary market incentives are. Companies competing to build useful, reliable, non-harmful products — following standard commercial pressures without any special coordination or regulation — end up solving alignment as a byproduct of building products people actually want to use. No government intervention is required.
3. Alignment by Default Government. Maybe market incentives alone aren't enough, but conventional policy interventions are. Governments applying familiar regulatory tools (liability law, safety standards, auditing requirements) in the ordinary way are sufficient to close the gap.. No unprecedented governance or coordination are required.

My rough credences:

1. Default Techniques sufficient: ~15%
2. Default Market sufficient (given training isn't): ~30%
3. Default Government sufficient (given market isn't): ~20%
4. Need something more unusual: ~35%

These are rough and the categories blur into each other, but the decomposition seems useful for locating where exactly you think the hard problem lies.

does anyone have takes on the "people should focus on their 25th percentile timelines rather than their median timelines" thing?

I mention cross-lab monitoring here:

Cross-lab monitoring - Each lab uses their own AI as the primary (e.g. Claude), and asks other labs to monitor the safety-critical activity (e.g. ChatGPT, Gemini, Llama, Grok). Furthermore, cross-lab monitoring would also mitigate AI-enabled coups. Note that cross-lab monitoring requires new security arrangements, such as provably-secure third-party servers.

The tech would need to be a bit messy because you'll need to send your competitor a "zero-knowledge proof" that you are using their AI only for monitoring catastrophic actions, and not for automating R&D, without leaking what your AIs are doing.

Potential setup:

1. Lab A sends their critical codebase to the third-party server
2. Lab B sends their monitor weights to the server, and instructs it to assist only with monitoring catastrophic actions
3. Lab A instructs B-monitor to flag any catastrophic actions

no one can see it other than the people you shared the doc with

Nice -- you probably win some sharpley for the interviewer's question

Please stop sharing google docs for comments

Instead: post the draft online, then share the link so people can comment in public.

I only share a google doc is if there's a specific person whose comments I want before posting online. But people often share these google docs in big slack channels —  at that point, just post online!

I think it slows innovation.

MMLU knowledge, fit well enough, requires inventing the universe

I suspect "fit well enough" doesn't track anything in reality.

Jerry Wei writes:

We expect this to become even more of an issue as AIs increasingly use tools to do their own research rather than rely on their learned knowledge (we tried to filter this kind of data as well, but it wasn't enough assurance against misuse).

I think his critique is this:

Suppose we had a perfect filtering system, such that the dangerous knowledge has zero mutual information with the model weights:

• Full degradation on WMDP, or whatever dangerous benchmark
• No degradation on MMLU, or whatever benign benchmark
• Scales to arbitrary sizes
• Robust to elicitation attacks, e.g. finetuning, prompting, etc

Nonetheless, the dangerous knowledge is "accessible" to the agent via web search + tools + in-context reasoning.

To solve this problem, we need either alignment techniques (e.g. train the model not to use these affordances) or inference-time monitoring techniques (e.g. constitutional classifiers). But if we had those techniques then we don't need the pretraining filtering.