1. Of course, the answer could be "A kajillion years from now" or "Never."
2. See this section of "Forecasting TAI with Biological Anchors" (Cotra (2020)) for a more full definition of "transformative AI."
3. I'm sorry. But I do think the rest of the series will be slightly more fun to read this way.
4. The examples here are of course simplified. For example, both Deep Blue and AlphaGo incorporate substantial amounts of "tree search," a traditionally-programmed algorithm that has its own "trial and error" process.
5. And they can include simulating long chains of future game states.
6. Some AIs could be used to determine whether papers are original contributions based on how they are later cited; others could be used to determine whether papers are original contributions based only on the contents of the paper and on previous literature. The former could be used to train the latter, by providing a "That's correct" or "That's wrong" signal for judgments of originality. Similar methods could be used for training AIs to assess the correctness of papers.
7. E.g., https://openai.com/blog/improving-language-model-behavior/
8. Due to improvements in hardware and software.
9. It's even worse than spaghetti code.
10. More books: Human Compatible, Life 3.0, and The Alignment Problem.