This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I'm sharing here a rough draft of a new curriculum I've made, called Understanding Intelligent Agency. I've written a lot lately about how AI alignment should be trying to develop a new scientific paradigm; this curriculum is my attempt to point directly at what that new paradigm looks like.
I'm sharing this in draft form because I don't know how long it will take me to get it to a state where I'm happy to promote it widely (at the very least, I'll be offline for most of the next week). I'm hoping it can be useful to people before that; and please feel free to leave comments or suggestions for additional readings.
I wonder how one could make useful additions or point at relevant conjectures. For example, Yudkowsky's Value Theory, the idea that FDT alone produces the ethics-like result prohibiting us from looting the universe even of a paperclip optimizer or my take trying to suggest that beyond-FDT morals aren't THAT far from sociopolitics.