This is a special post for quick takes by Jesse Li. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Would an "Unsupervised Program for Alignment Research" be effective?
I think there may be a substantial pool of people trying to contribute to AI safety who are motivated, talented, and capable of doing good independent research, who nonetheless haven't been accepted to any existing mentored fellowship.
Synthesizing a few perspectives I've heard, which I don't think are necessarily contradictory:
Note: I'm a SPAR mentee and have only been on the applying side of these programs, so I don't know how accurate the above claims are. But they do seem to line up with what I've seen mentors say.
So, what if we just found these people & gave them funding anyway, to do independent research?
I think finding ways to scale up mentorship better is a good idea, but I worry that your proposal is mostly just SPAR. The mentors already aren't expected to do research and technically only have to have a research question and meet with mentees every week or so. Even with that, my SPAR mentor spends a significant amount of time explaining which aspects of the research question are interesting, where other people will push back, and what's happening in the field that affects what we should be doing.
It does feel like there should be ways to scale this further to reduce the demands on mentors though. Part of the complexity seems to be from the size of projects. A set of micro-experiments that mentees should run and then write up would be interesting (although it might be hard to find something small enough that Claude can't do on its own).
Valid concern; my proposal is literally just putting "un-" in front of "Supervised Program for Alignment Research". In order to do something SPAR isn't already, I think you'd have to reduce mentor involvement requirements by an order of magnitude:
This would probably result in a reduction in research quality but the increased scalability might make up for it. Also, I'm assuming that SPAR and other fellowships are bottlenecked by mentors available to dedicate enough time.
Canary GUIDs are often attached to published AI safety work that's sensitive (e.g. particular misalignment detection approaches) because we don't want future generations of (potentially misaligned) models to learn about them and know how to scheme around them.
But even if labs followed best practices and filtered out all their crawled documents with that GUID, it's not enough - firstly, papers can get summarized, reworded, and republished without the canary; secondly, if the researchers used coding agents to develop that work, there's a lot more detailed work in transcripts which probably don't have the canary.
How can we counteract this? Perhaps it'd be good to have an organization whose purpose is to collect a database of sensitive documents (e.g. with the canary) and provide a programmatic interface to evaluate how close a given piece of text is semantically to a sensitive idea. You could use this service for training data filtering, as well as experiments on unlearning.
A crazy idea, which to be honest I haven't fully thought through - would it be possible and net beneficial to intentionally release an AI model (~Mythos capability) whose primary objective was to infect data centers, self replicate, and cause controlled disruption (a la ransomware), solely as an artificial warning shot for runaway AGI?
Benefits (assuming widespread disruption is caused, e.g. large chunks of the internet goes down):
Costs:
(Vaguely related/source of inspiration: Tom Scott's "The (fictional) day when Google forgot to check passwords" from a decade ago, hypothesizing what would happen in a large scale cybersecurity failure)
[Aside - I'm a new contributor. The guidelines mention a high bar with respect to AI content and suggest first posting to the AI Questions Open Thread, but is that still active and correct advice? The last open thread seems to be 3 years ago, but it seems like they're supposed to happen monthly?]