One of the most valuable things mathematicians can do in this moment is to engage with AI-generated proofs and disentangle what techniques were used and whether it involved anything novel that is not a combination and stacking of existing techniques. This is valuable information for alignment difficulty and AI timelines.
That is, it is simply not enough to flippantly say “lol now this wasn’t in the training data huh!” It’s incredibly frustrating to me how incurious people can be about the specifics of the situation because they prefer the easy dunks against the loudest skeptics.
The contrary also applies. It could be the case that in order for AI to be transformative (or even capable of takeover) it just needs to "stack existing techniques" fast enough. Maybe we are overestimating AI generalisation, but maybe we underestimating reality's homogeneity.
(yes I posted this under your tweet as well)
Thank you for your reply. I assume you didn’t mean it this way, but it is also a fairly common objection I hear when someone even questions AI capabilities. Yet I often get the impression that it serves as a conversation stopper rather than an actual attempt at understanding how true it is.
If it’s true the AI models are limited in the types of cognitive moves available to them, they still solved RH. Of course! But I still think we shouldn’t stop, again, playing out the details. It may be that you can get AIs that can build Dyson spheres without strong OOD ... (read more)
I think this is true, but I'd be predicting longer timelines if this is the case.
More specifically, my median for transformative milestones would move from the 2027-2030 to 2035-2040, and conditional on only weak regulation passing on AI, would predict a 90-99% chance of superintelligence by 2045-2050.
The reason is that AIs are just good enough in the early 2030s to reproduce themselves via robotics surprisingly fast, and compute investments will partially shift to robot investment, and the pace of model improvement slowly increases such that the breakthroughs necessary eventually happen through sheer brute force/the sea of AIs eventually drown the hard problems through numbers.
In a way, it recapitulates how humans eventually managed to take over the planet, but the reasons for this happening in AI is substantially different than why humans had to do it:
20 years from RSI to takeoff.