Thanks for the heads up. Can you share which AI models were involved?

Maybe we want a multi-level categorization scheme instead? Something like:

Level 0: Author completely abstains from LLM use in all contexts (not just this post)
Level 1: Author uses LLMs but this particular post was made with no use of LLM whatsoever
Level 2: LLM was used (e.g. to look up information), but no text/images in the post came out of LLM
Level 3: LLM was used for light editing and/or image generation
Level 4: LLM was used for writing substantial parts
Level 5: Mostly LLM-generated with high-level human guidance/control/oversight

It occurred to me that one major success story of banning a technology (whether justifiably or not), is the laws against genetic engineering and cloning in humans. It makes me wonder what can we learn from that, which is applicable to promoting a global moratorium on ASI.

10 years ago I argued that approval-based AI might lead to the creation of a memetic supervirus. Relevant quote:

Optimizing human approval is prone to marketing worlds. It seems less dangerous than physicalist AI in the sense that it doesn't create incentives to take over the world, but it might produce some kind of a hyper-efficient memetic virus.

I don't think that what we see here is literally that, but the scenario does seem a tad less far-fetched now.

Fixed!

I did link the relevant section of my agenda post:

This work is my first rigorous foray into compositional learning theory.

A brief and simplified summary:

• In order to have powerful learning algorithms with safety guarantees, we first need learning algorithms with powerful generalization guarantees that we know how to rigorously formulate (otherwise how do you know the algorithm will correctly infer the intended goal/behavior from the training data?).
• Additionally, in order to formally specify "aligned to human values", we need to formally specify "human values", and it seems likely that the specification of "X's values" should be something akin to "the utility function w.r.t. which X has [specific type of powerful performance guarantees]". These powerful performance guarantees are probably a form/extension of powerful generalization guarantees.
• Both reasons require us to understand the kind of natural powerful generalization guarantees that efficient learning algorithms can satisfy. Moreover, such understanding would likely be applicable to deep learning as well, as it seems likely deep learning algorithms satisfy such guarantees, but we currently don't know how to formulate them.
• I conjecture that a key missing ingredient in deriving efficient learning algorithms with powerful guarantees (more powerful than anything we already understand in computational learning theory), is understanding the role of compositionality in learning. This is because compositionality is a ubiquitous feature of our thinking about the world and, intuitively, particular forms of compositionality are strong candidates for properties that are both very general and strong enough to enable efficient learning. This line of thinking led me to some success in the context of control theory, which is a necessary ingredient of the kind of guarantees we will ultimately need.
• I identified sequence prediction / online learning in the deterministic realizable case as a relatively easy (but already highly non-trivial) starting point for investigating compositional learning.
• For the reasons stated in the OP, this led me to ambiguous online learning.

I'm open to chatting on Discord.

I never did quite that thing successfully. I did have one time when I dropped progressively unsubtle hints on a guy, who remained stubbornly oblivious for a long time until he finally got the message and reciprocated.

Hold my beer ;)