Lots of good information - for people considering joining a nonprofit board and reading this, be sure to ask about whether the org carries directors and officers (often referred to as D&O) insurance - otherwise you could be personally liable for problems that happen!

Interesting post, and I generally agree.

One note - you appear to be quoting David Chapman, not Yudkowsky. The Twitter post you linked to was written by Chapman. It's also not exactly what the tweet says. Can you maybe update to reflect that it's a Chapman quote, or directly link to where Yudkowsky said this? Apologies if I'm missing something obvious in the link.

We were surprised to find a decrease in publications on the arXiv in recent years, but identified the cause for the decrease as spurious and fixed the issue in the published dataset (details in Fig. 4).

I'd be interested in hearing more about how the decrease was determined to be spurious; I looked at Fig. 4 but am not understanding how that decision was made based on the figure, if that was the intention.

This is great, thanks so much for pulling this together (and for linking to our Gato explainer!)

It just so happens I'm working with a group of people through the Cambridge EA Technical AI alignment curriculum, and this idea of IDA is what week 5 is all about - lots of further reading for those who want.

One prompt in the weekly curriculum asks whether there are any tasks that cannot easily be broken down in the way described above, and therefore might not be useful for IDA. One thing I can thing of offhand is large leaps in scientific understanding. For example, if you took 20 physicists and gave them the problems of the day, it's not clear that they ever would have come up with Einstein's theory of relativity. Given that problem, I wonder what the implications are for trying to use IDA to create AGI - does this mean there are certain types of tasks that a IDA-based AGI will not be so good at?

Thanks for reaching out! At the moment it's only here and on the EA forum; we may be publishing it elsewhere but that's still in the works. If you want to PM me with your request I'll see what I can do!