This is a special post for quick takes by Kabir Kumar. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
If you haven't spent more than 10 minutes of non cached thinking about what would see differently in the world if we were less than 2 years to RSI (actual intelligence explosion that results in ASI being on Drexler style nanocomputers or something even more efficient within 3 weeks), please, please do so. If you haven't done so within the last 3 months and then tried to search for things close to your predictions, please please do so
It seems to me that there must be a limit of capabilities per FLOP, it's just a question of how far away we are from those limits with our current models. I don't think we can possibly know how close we are except by "upper bound seen so far" and weak evidence based on past algorithmic improvements.
2JBlack
Having given it quite a lot more than 10 minutes of non-cached thinking over the past months, I don't believe that there can be any predictable differences between being "a year or two before a weeks-scale takeoff" and "partway through an S-curve over a scale of years or slower".
We don't know whether massively better AI is possible or not on current or near-future hardware, and the only plausible way to find out is if it actually happens. Our AI systems are now just starting to explore regions of cognitive capability space that we have no experience with, and we don't and can't know whether the self-improvement returns are faster or slower than linear with capability. We don't know which of the problems we've experienced with progress in nanotechnology (or other potential advanced computing substrates) are solvable with more intelligence, and which are fundamental to physical nature.
We basically can't know any of these things without either being far more intelligent ourselves, or actually testing these things in the real world.