This retrospective focuses on the 4-month MATS extension phase (referred to as "MATS 5.1") that ran from April 1 to July 25, 2024, and presents findings gathered from an end-of-extension survey as well as follow-up interviews and surveys ~5 months after the program.
Main changes from the 4.1 to 5.1 extension phase:
Key takeaways from MATS extension impact:
I think this was good, but I really think laying out the basic arguments for convergent instrumental goals is a foundational part of introducing the topic to a new audience (which I expect is the case for most of Lex's listeners) which wasnt sufficiently explained.
Making it clear that most innocuous goals beget resource acquisition and self preservation which is what puts an agentic AI in conflict with humans by default is what really makes the concern concrete for many people. Otherwise I think there is a tendancy to assume that some leap is required for a system to be in conflict which is much harder to picture and seems intuitively more preventable.
Sorry I agree that comment and those links left some big inferential gaps.
I believe the link below is more holistic and doesn't leave such big leaps (admittedly it does have some 2021-specific themes that haven't aged so well, but I don't believe they undermine the core argument made).
https://www.fabricatedknowledge.com/p/the-rising-tide-of-semiconductor
This still leaves a gap between cost per transistor and overall compute cost, but that's a much smaller leap e.g. frequency being bound by physical constraints like the speed of light. etc..
To evidence my point about this trend getting even worse after 2030 - EUV lithography was actively being pursued for decades before active usage in 2030. My understanding is that we don't have anything that significant at the level of maturity that EUV was at in the 90s. Consider my epistemic status on this point fairly weak though.
Excellent post, thank you. I look forward to playing with the notebook model.
One observation on the timelines (which I realise is somewhat tangential to the core content of the post) is that I believe your estimates around compute cost scaling feel quite optimistic to me.
Memory and compute density scaling is already dropping off meanwhile everyone seems to be struggling with yields on sub-3nm nodes despite huge spend, exacerbating the problem for cost scaling.
https://fuse.wikichip.org/news/7343/iedm-2022-did-we-just-witness-the-death-of-sram/
https://www.semianalysis.com/p/tsmcs-3nm-conundrum-does-it-even
We don't really have many more promising technologies in the pipeline to radically get cost per transistor down so I expect improvements to slow quite a bit towards the end of this decade.