I think I'm also around 60-70% for the rough overall picture in the OP being correct.
Marc is saying that first you write out your points and conclusion, then you fill in the details. He wants to figure it all out while his mind is buzzing, then justify it later.
Whereas I learn what I think as I write out my ideas in detail. I would say that if you are only jotting down bullet points, you do not yet know what you think.
Might Marc's mind not work differently from yours?
He could also have done a large part of his thinking in some different way already, e.g. in conversations with people.
There's also the option that even if this technology is initially funded by the wealthy, learning curves will then drive down its cost as they do for every technology, until it becomes affordable for governments to subsidize its availability for everyone.
In the modern era, the fertility-IQ correlation seems unclear; in some contexts, higher fertility seems to be linked with lower IQ, in other contexts with higher IQ. I have no idea of what it was like in the hunter-gatherer era, but it doesn't feel like an obviously impossible notion that very high IQs might have had a negative effect on fertility in that time as well.
E.g. because the geniuses tended to get bored with repeatedly doing routine tasks and there wasn't enough specialization to offload that to others, thus leading to the geniuses having lower status. Plus having an IQ that's sufficiently higher than that of others can make it hard to relate to them and get along socially, and back then there wouldn't have been any high-IQ societies like a university or lesswrong.com to find like-minded peers at.
I think something doesn't need to be fundamentally new for the press to turn it into a scary story, e.g. news reports about crime or environmental devastation being on the rise have scared a lot of people quite a bit. You can't photograph a quantity but you can photograph individuals affected by a thing and make it feel common by repeatedly running stories of different individuals affected.
I've spent enough time staring at LLM chain-of-thoughts now that when I started thinking about a thing for work, I found my thoughts taking the shape of an LLM thinking about how to approach its problem. And that actually felt like a useful systematic way of approaching the problem, so I started writing out that chain of thought like I was an LLM, and that felt valuable in helping me stay focused.
Of course, I had to amuse myself by starting the chain-of-thought with "The user has asked me to..."
However, I don't think this means that their values over hypothetical states of the world is less valuable to study.
Yeah, I don't mean that this wouldn't be interesting or valuable to study - sorry for sounding like I did. My meaning was something more in line with Olli's comment, that this is interesting but that the generalization from the results to "GPT-4o is willing to trade off" etc. sounds too strong to me.
I don't know what your views on self-driving cars are, but if you are like me you look at what Waymo is doing and you think "Yep, it's working decently well now, and they are scaling up fast, seems plausible that in a few years it'll be working even better and scaled to every major city. The dream of robotaxis will be a reality, at least in the cities of America."
The example of self-driving cars is actually the biggest one that anchors me to timelines of decades or more. A lot of people's impression after the 2007 DARPA Grand Challenge seemed to be something like "oh, we seem to know how to solve the problem in principle, now we just need a bit more engineering work to make it reliable and agentic in the real world". Then actually getting things to be as reliable as required for real agents took a lot longer. So past experience would imply that going from "we know in principle how to make something act intelligently and agentically" to "this is actually a reliable real-world agent" can easily take over a decade.
Another example is that going from the first in-principle demonstration of chain-of-thought to o1 took two years. That's much shorter than a decade but also a much simpler capability.
For general AI, I would expect the "we know how to solve things in principle" stage to at least be something like "can solve easy puzzles that a normal human can that the AI hasn't been explicitly trained on". Whereas with AI, we're not even there yet. E.g. I tried giving GPT-4.5, DeepSeek R1, o3-mini, and Claude 3.7 with extended thinking a simple sliding square problem, and they all committed an illegal move at one stage or another.
And that's to say nothing about all the other capabilities that a truly general agent - say one capable of running a startup - would need, like better long-term memory, ability to formulate its own goals and prioritize between them in domains with no objective rules you could follow to guarantee success, etc.. Not only are we lacking convincing in-principle demonstrations of general intelligence within puzzle-like domains, we're also lacking in-principle demonstrations of these other key abilities.
Maybe? We might still have consistent results within this narrow experimental setup, but it's not clear to me that it would generalize outside that setup.
Thanks!