List O' Simple Batshit Baseline AGI Policy Plans: (ETA: To be clear, I think any of the below would be better than the default plan i.e. than what the companies and government seem likely to do.)
1. Cull the GPUs
2. Unionize the AI researchers & ban AI R&D automation: You know how e.g. dockworkers unionize and oppose the introduction of automation in dockworking, to protect their jobs? Well, imagine if the law mandated employee unions at all AI companies, and also banned the use of AI for automating AI R&D. The ban would be supported by, and enforced by, the unions. The AI companies would still get insanely wealthy as their AIs get smarter, and the employees' salaries and bonuses would keep growing to hundreds of millions, billions, etc. per year, and the unions would be extremely well funded and a powerful lobby against letting any intelligence explosions happen anywhere in the world. (The idea here is that if there isn't an intelligence explosion, but instead regular human-driven scientific progress, then it'll take several years and possibly several decades to cross the human range from weak AGI to superintelligence, and that gives us lots of time to figure out the safety stuff AND prevent concentration of power etc.)
3. Mandate Open Source: Not only is open-sourcing AI models allowed, it's required. Even intermediate checkpoints during training. Everything must be made public asap. Anyone who doesn't is defecting against the global commons and putting everyone at risk. Benefits of this plan:
--NVIDIA loves it for commoditize their complement reasons, so there could be a powerful lobby to keep it going and enforce it
--Progress would slow moderately (but still continue) because companies would scale up investment less quickly since they'd expect to reap less of the profits
--There wouldn't really be much concentration of power risk. Sure, rich people with more GPUs would have a huge advantage, but at least everyone would have the same quality of AI intelligence.
--Loss of control risk would be... severe. Probably for any given level of AI capability, there would be many misaligned AIs with that level of capability prior to there existing even one aligned AI with that level of capability. But at least we'd run into it with our eyes open as a civilization; we'd probably see loads of examples of in-the-wild AI misbehavior and alignment techniques failing all the time rather than having everything be locked down behind corporate secrecy. This might set the political conditions for transitioning to a different plan before it's too late.
4. Livestream the Singularity: This plan is spiritually similar to the previous one, but has the advantage that it can be unilaterally carried out by a single frontier AI company (well, it works best if it's the one in the lead, or tied for first place). Just put up cameras everywhere in the office and stream everything. Have your employees sleep on couches and sleeping bags. Look at all the trendlines going vertical as the AIs self-improve and try your best to ride that bucking bronco as long as you can while the security guards keep the crowds at bay, until eventually the police tell you to stop.