more robust definitions of what it means for an ai system to be safe / safe enough seem really useful for my thought process and also for getting better consensus but i'm not sure how you'd start to define one
You'd prefer the world where the system exists, over the world where it does not? (If the "you" who makes the judgment was omniscient, but the "you" who exists in the world with the system is just the normal you.)