agreed that among all paths to good things that I see, a common thread is somehow uplifting human cognition to keep pace with advanced AI. however, I doubt that that's even close to good enough - human cooperation is shaky and unreliable. most humans who think they'd do good things if made superintelligent probably are wrong due to various ways to value drift when the structure of one's cognition changes, and many humans who say they think they'd do good things are simply lying, rather than deluding themselves or overestimating their own durable-goodness. it seems to me that in order to make this happen, we need to make AIs that strongly want all humans and humanity and etc emergent groups to stick around, the way a language model wants to output text.

ignoring whether anthropic should exist or not, the claim

successful alignment work is most likely to come out of people who work closely with cutting edge AI and who are using the modern deep learning paradigm

(which I agree with wholeheartedly,)

does not seem like the opposite of the claim

there was no groundbreaking safety progress at or before Anthropic

both could be true in some world. and then,

pragmatic approaches by frontier labs are very unlikely to succeed

I believe this claim, if by "succeed" we mean "directly result in solving the technical problem well enough that the only problems that remain are political, and we now could plausibly make humanity's consensus nightwatchman ai and be sure it's robust to further superintelligence, if there was political will to do so"

but,

alternative theoretical work that is unrelated to modern AI has a high chance of success

I don't buy this claim. I actually doubt there are other general learning techniques out there in math space at all, because I think we're already just doing "approximation of bayesian updating on circuits". BUT, I also currently think we cannot succeed (as above) without theoretical work that can get us from "well we found some concepts in the model..." to "...and now we have certified the decentralized nightwatchman for good intentions sufficient to withstand the weight of all other future superhuman minds' mutation-inducing exploratory effort".

I claim theoretical work of relevance needs to be immediately and clearly relevant to deep learning as soon as it comes out if it's going to be of use. Something that can't be used on deep learning can't be useful. (And I don't think all of MIRI's work fails this test, though most does, I could go step through and classify if someone wants.)

I don't think I can make reliably true claims about anthropic's effects with the amount of information I have, but their effects seem suspiciously business-success-seeking to me, in a way that seems like it isn't prepared to overcome the financial incentives I think are what mostly kill us anyway.

zooming out as far as it goes, the economy is guaranteed to become decreasing-returns-to-scale (upper-bounded returns to scale) once grabby alien saturation is reached and there is no more unclaimed space in the universe

points Id want to make in a main post.

• you can soonishly be extremely demanding with what you want to prove, and then ask a swarm of ais to go do it for you. if you have a property that you're pretty sure would mean your ai was provable at being good in some way if you had a proof of your theorem about it, but it's way too expensive to find the space of AIs that are provable, you can probably combine deep learning and provers somehow to get it to work, something not too far from "just ask gemini deep thinking", see also learning theory for grabbing the outside and katz lab in israel for grabbing the inside.
• probably the agent foundations thing you want, once you've nailed down your theorem, will give you tools for non-provable insights, but also a framework for making probability margins provable (eg imprecise probability theory).
  • you can prove probability margins about fully identified physical systems! you can't prove anything else about physical systems, I think?
  • (possibly a bolder claim) useful proofs are about what you do next and how that moves you away from bad subspaces and thus how you reach the limit, not about what happens in the limit and thus what you do next
• finding the right thing to prove is still where all the juice is, just like it was before. but I'd be writing to convince people who think proof and theory is a dead end that it's not because actually bitter lesson can become sweet lesson if you have something where you're sure it's right in the sense that more optimization power is more goodness. proofs give you a framework for that level of nailing down. you can eg imagine using a continuous relaxation of provability as your loss function
• so I'd want to argue that the interesting bit is how you prove that your reinforcement learning process will always keep coming back for more input, without breaking your brain, and will keep you in charge. the main paths I see for this making sense is a cev thing with a moment in time pinned down as the reference point for where to find a human, that's the PreDCA/QACI style approach; or, jump past CEV and go for encoding empowerment directly, and then ask for empowerment and non-corruption of mindlike systems or something.
• this is sort of an update of the old miri perspective, and it ends up calling for a bunch of the same work. so what I'd be doing in a main post is trying to lay out an argument for why that same work is not dead, and in fact is a particularly high value thing for people to do.
• I'd be hoping to convince noobs to try it, and for anthropic and deepmind to look closer at what it'd take to get their AIs to be ready to help with this in particular.
• I'm posting this instead of a main post because I've developed an ugh field around writing main post, so I need to make it ok to post sloppy versions of the point first and get incremental bite from people thinking it sounds unconvincing, rather than trying to frontload the entire explanatory effort before I even know what people think sounds implausible.

seems like if it works to prevent ASI-with-10yr-planning-horizon-bad-thing, it must also work to prevent waterworld rl with 1 timestep planning horizon-bad-thing.

• if you can't mechinterp tiny, language-free model, you can't mechinterp big model (success! this bar has been passed)
• if you can't prevent emergent scheming on tiny, language-free model, you can't prevent emergent scheming on big model
• as above for generalization bounds
• as above for regret bounds
• as above for regret bounds on CEV in particular*

if you can do it on small model doesn't mean you can do it on big model. but a smoke test is that your method had better work on a model so small it can't talk. if your method requires a model big enough to talk in order to do anything that seems promising, you probably aren't robust against things that would arise even without the presence of language, and you might be getting punked by the "face".

* (which I call out because CEV is often treated as a thing where you need an enormous model to make any statement about it, which I think is importantly misleading, because I think if a CEV-correctness certification is ever going to be possible, it should be one that monotonically improves with scale, but produces only-finitely-bad bounds on small model)

some people (not on this website) seem to say "existential risk" in ways that seem to imply they don't think it means "extinction risk". perhaps literally saying "extinction risk" might be less ambiguous.

which is exactly what we're worried about from AI, and is why I don't think this is an AI-specific problem, it's just that we need to solve it asymptotically durably for the first time in history. I'm having trouble finding it right now, but there was a shortform somewhere - I thought it was by vanessa kosoy, but I don't see it on her page; also not wentworth. IIRC it was a few days to weeks after https://www.lesswrong.com/posts/KSguJeuyuKCMq7haq/is-vnm-agent-one-of-several-options-for-what-minds-can-grow came out - about how the thing that forces being a utility maximizer is having preferences that are (only?) defined far out in the future.

To be clear, I am in fact saying this means I am quite concerned about humans whose preferences can be modeled by simple utility functions, and I agree that money and living forever are two simple preferences where, if they're your primary preference, you'll probably end up looking relatively like a simple utility maximizer.

benchmarks are how people optimize for capabilities. working on them has been a core method of increasing capabilities over the past 10+ years of machine learning, at minimum, since I started paying attention in 2015. if you want to increase endgame (asi) alignment outcomes, measuring capabilities is a distraction at best.

edit: whether this applies to bad behavior benchmarks is contested below. I happily restrict my claim to benchmarks that are measuring things that labs would have some reason to optimize for and still believe that covers this topic.

My current view is that alignment theory should work on deep learning as soon as it comes out, if it's the good stuff, and if it doesn't, it's not likely to be useful later unless it helps produce stuff that works on deep learning. Wentworth (and now Condensation), SiLT, and Causal Incentives are the main threads that already seem to have achieved this somewhat; I'm optimistic Ngo is about to. DEC seems potentially relevant. (list edited 4mo later, same entries but improved ratings.)

I'll think about your argument for mechinterp. If it's true that the ratio isn't as catastrophic as I expect it to turn out to be, I do agree that making microscope AI work would be incredible in allowing for empiricism to finally properly inform rich and specific theory.