If the pie is bigger, the only possible problem is bad politics. There is no technical AI challenge here. There might be a technical economical problem. It’s anyhow unrelated to the skill set of AI people. Bundling is not good, and this article bundles economic and political problems into AI alignment.

Edge AI is the only scenario where AI can self replicate and be somewhat self sufficient without a big institution though? It’s bad for AI dominion risk, good for political centralization risk.

I’ve long taken to using GreaterWrong. Give it a try, lighter and more featureful.

But the outside view on LLM hitting a wall and being a “stochastic parrot” is true? GPT4O has been weaker and cheaper than GPT4T in my experience, and the same is true w.r.t. GPT4T vs. GPT4. The two versions of GPT4 seem about the same. Opus is a bit stronger than GPT4, but not by much and not in every topic. Both Opus and GPT4 exhibit patterns of being a stochastic autocompleter, and not a logician. (Humans aren’t that much better, of course. People are terrible at even trivial math. Logic and creativity are difficult.) DallE etc. don’t really have an artistic sense, and still need prompt engineering to produce beautiful art. Gemini 1.5 Pro is even weaker than GPT4, and I’ve heard Gemini Ultra has been retired from public access. All of these models get worse as their context grows, and their grasp of long range dependencies is terrible.

The pace is of course still not too bad compared with other technologies, but there doesn’t seem to be any long-context “Q*” GPT5s in store, from any company.

PS: Does lmsys do anything to control for the speed effect? GPT4O is very fast, and that alone should be responsible for many ELOs.

Can you create a podcast of posts read by AI? It’s difficult to use otherwise.

Can you create a podcast of posts read by AI? It’s difficult to use otherwise.

I doubt this. Test-based admissions don't benefit from tutoring (in the highest percentiles, compared to less hours of disciplined self-study) IMO. We Asians just like to optimize the hell of them, and most parents aren't sure if tutoring helps or not, so they register their children for many extra classes. Outside of the US, there aren't that many alternative paths to success, and the prestige of scholarship is also higher.

Also, tests are somewhat robust to Goodharting, unlike most other measures. If the tests eat your childhood, you'll at least learn a thing or two. I think this is because the Goodharting parts are easy enough that all the high-g people learn them quickly in the first years of schooling, so the efforts are spent just learning the material by doing more advanced exercises. Solving multiple-choice math questions by "wrong" methods that only work for multiple-choice questions is also educational and can come in handy during real work.

AGI might increase the risk of totalitarianism. OTOH, a shift in the attack-defense balance could potentially boost the veto power of individuals, so it might also work as a deterrent or a force for anarchy.

This is not the crux of my argument, however. The current regulatory Overton window seems to heavily favor a selective pause of AGI, such that centralized powers will continue ahead, even if slower due to their inherent inefficiencies. Nuclear development provides further historical evidence for this. Closed AGI development will almost surely lead to a dystopic totalitarian regime. The track record of Lesswrong is not rosy here; the "Pivotal Act" still seems to be in popular favor, and OpenAI has significantly accelerated closed AGI development while lobbying to close off open research and pioneering the new "AI Safety" that has been nothing but censorship and double-think as of 2024.

A core disagreement is over “more doomed.” Human extinction is preferable to a totalitarian stagnant state. I believe that people pushing for totalitarianism have never lived under it.