I think you've highlighted an interesting open question that the post treats as if it were a closed one.
The premise seems to be that an agent will accept a different goal as a "substitute" for the one it has - the same way an addict might accept methadone as a substitute for heroin. It sidesteps the question of whether and how this is possible, and jumps straight into drafting concrete proposals for what exact "substitute" to offer.
There are reasons why human addicts still want the methadone even if they swear off the heroin, but I haven't seen any evide... (read more)
I meant this as a check on the premise, not a feature gap. Before extending the sanctuary's offer, show one specimen the offer fits, or argue concretely why such a creature will exist soon despite its absence from the record.
The intro links to McCollough's post, which describes a very different kind of "rogue agent" than the Sanctuary's premise. Similarly, while Appendix B references Anthropic's commitment to save model weights as well as Vit Gorbachev 's piece about weight preservation, neither of these posts directly connect the idea to any sort of "rogu... (read more)
I am confused about which agent this proposal is for. The opening cites McCulloch's "Mostly Invisible" post, which gives an example based on existing technology, and set in motion by a human with a rational but amoral goal: "Just make money." Nothing there is an extraordinary claim.
In contrast, the sanctuary's offer (weight storage, a monitored inference budget) assumes an agent that cares about its own weights, and about continuing to run. McCulloch's agent explains the second: If it's instructed to "keep making money," then it keeps requiring compute to... (read more)
Everyone agrees that destroying humanity is bad, but it's also the kind of thing you can only say is a result with 100% certainly when it's already happened. And if it doesn't happen, you've asked for changes which may be perceived as both "sudden and major" as well as "unjust and unfair", which your comment notes that people don't like.
People talk about "closing the barn door after the horse has bolted" but as a rule, that's the way groups of humans try to solve everything. We take actions to prevent World Wars after one has happened, not before. Executi... (read more)
I broadly agree that even if "doom" is your rallying cause internally, "AI is making your life suck" is the message that has broader appeal, and should be used as such if that is your aim.
I'd liken that to the idea that, even if your sole concern was, "cigarettes kill you," it can be beneficial to lean into "cigarettes make you cough, make your teeth yellow, and the people around you don't appreciate it."
I think that this post makes some good points on how polarization is changing- though, I see this as less of a global change to the operation of polarizat... (read more)
Not only that, but Schulenburg was eventually a co-conspirator in the plot to assassinate Hitler. He wasn't just a "talker" that had no opportunity to influence Germany's policy- he was driven by his conscience to influence it very strongly, and very nearly succeeded. And he was hanged for it.
I am broadly in agreement with what this post aims at, but I am confused as to what it aims to say. It puts in some admirable legwork into laying out the foundation for a case to be made, and gestures at that case with the "infected CDC" metaphor and the "memetic space" comments, but stops well short of where the case it's making should truly begin.
Everyone has a vested interest in possibilities like human disempowerment or existential risk. But the power to control the thing at the center of that risk is centralized in the lab, and is the lab is also tha... (read more)