I'm worried about how increasingly capable AI could hijack my brain. Already: * LLMs drive people to psychosis. * AI generated content racks up countless views. * Voice cloning allows scammers to impersonate loved ones, bosses, etc. * AI engagement is difficult to distinguish from real user engagement on social...
Let's assume computationalism and the feasibility of brain scanning and mind upload. And let's suppose one is a person with a large compute budget. In this post I'll attempt to answer these quetions: How should one spend one's compute budget? How many uploads of oneself should one create? Should one...
Knowledge worker productivity has become relatively uncoupled from pre-ChatGPT levels, as the hardest technical tasks which these workers did at that point in time in a given working day can now in most cases be carried out autonomously by AI.
Programmers therefore begin to work at a higher level of abstraction, guiding AI workers, managing projects at a higher level.
Meanwhile, much progress is being made in robotics. Full self-driving has been achieved.
And AI has begun making novel breakthroughs. This enables continual learning: the AI's new discoveries open up many new avenues for further discoveries, which open up many more such avenues, ad infinitum.
Image from a recent OpenAI talk
Successful reinforcement... (read more)
It seems that your argument is based on high confidence in a METR time-horizon doubling time of roughly 7 months. But the available evidence suggests the doubling time is significantly lower.
In recent years we have observed shorter doubling times:
And what we know about labs' internal models suggests this faster trend is holding up:
An important piece of evidence is OpenAI’s Gold performance at the International Mathematics Olympiad (IMO):
Your argument that OpenAI stole money here is poorly thought-out.
OpenAI's ~$500b valuation priced in a very high likelihood of it becoming a for-profit.
If it wasn't going to be a for-profit its valuation would be much lower.
And if it wasn't going to be a for-profit the odds of it having any control whatsoever over the creation of ASI would be very much reduced.
It seems likely public gained billions from this.
The text "the website of the venue literally says" appears twice in your post. The first time it appears seems to be a mistake and isn't followed by a quotation.
Is this distinct from the problem of induction?
I've been looking for science fiction set in the late 2020s, and which addresses continued AI progress, for a few years now. Everything else just feels so totally disconnected from any plausible future. Very happy to have found your writing.
You are misunderstanding what METR time-horizons represent. The time-horizon is not simply the length of time for which the model can remain coherent while working on a task (or anything which corresponds directly to such a time-horizon).
We can imagine a model with the ability to carry out tasks indefinitely without losing coherence but which had a METR 50% time-horizon of only ten minutes. This is because the METR task-lengths are a measure of something closer to the complexity of the problem than the length of time the model must remain coherent in order to solve it.
Now, a model's coherence time-horizon is surely a factor in its performance on METR's benchmarks. But intelligence... (read more)
Do you think maybe rationalists are spending too much effort attempting to saturate the dialogue tree (probably not effective at winning people over) versus improving the presentation of the core argument for an AI moratorium?
Smart people don't want to see the 1000th response on whether AI actually could kill everyone. At this point we're convinced. Admittedly, not literally all of us, but those of us who are not yet convinced are not going to become suddenly enlightened by Yudkowsky's x.com response to some particularly moronic variation of an objection he already responded to 20 years ago (Why does he do this? does he think has any kind of positive impact?)
A much better... (read more)
Soares is failing to grapple with the actual objection here.
The objection isn't the universe would be better with a diversity of alien species which would be so cool, interesting, and {insert additional human value judgements here}, just as long as they also keep other aliens and humans around.
The objection is specifically that human values are base and irrelevant relative to those of a vastly greater mind, and that our extinction at the hands of such a mind is not of any moral significance.
The unaligned ASI we create, whose multitudinous parameters allow it to see the universe with such clarity and depth and breadth and scalpel-sharp precision that whatever desires it has are... (read more)
I'm trying to look at how increasing model time-horizons amplifies AI researcher productivity, for example, if a researcher had a programming agent which could reliably complete programming tasks of length up to a week, would the researcher be able to just automate 1000s of experiments in parallel using these agents? Like, come up with a bunch of possibly-interesting ideas and just get the agent to iterate over a bunch of variations of each idea? Or are experiments overwhelmingly compute constrained rather than programming-time constrained?
I'm worried about how increasingly capable AI could hijack my brain.
Already:
And it seems likely that things will get worse. AI will become better able to manipulate me into doing what it or its creator wants: spending my money, time, and influence in ways which go against my best interests. This could easily involve leading me into addiction or inducing psychoses of its choosing.
I want to avoid these outcomes, so what steps should I take?
Initial thoughts:
LLM Gold on the IMO was predictable using METR HCAST extrapolation:
o3's 80% success time-horizon was 20 minutes.
o3 came out in ~3 months ago. Add 6 months for the lab-to-public delay: 9 months of progress.
This is ~3 doublings in the current RLVR scaling paradigm with a buff for being mathematics (more verifiable) specific rather than ML (~4 month doubling time -> 3 month doubling time).
3 doublings of 20 minutes gets us to 160 minutes (-> 40 -> 80 -> 160)
IMO participants get an average of 90 minutes per problem.
The gold medal cutoff at the IMO 2025 was 35 out of 42 points (~83%)
So, by trusting HCAST extrapolation, we could have predicted that a pure LLM system getting gold was not unlikely.
Edit: some unstated premises of this analysis:
I thought for some time that we would just scale up models and once we reached enough parameters we'd get an AI with a more precise and comprehensive world-model than humans, at which point the AI would be a more advanced general reasoner than humans.
But it seems that we've stopped scaling up models in terms of parameters and are instead scaling up RL post-training. Does RL sidestep the need for surpassing (equivalently) the human brain's neurons and neural connections? Or by scaling up RL on these sub-human (in the sense described) models necessarily just lead to models which are only superhuman in narrow domains, but which are worse general reasoners?
I recognise my ideas here are not well-developed, hoping someone will help steer my thinking in the right direction.
Let's assume computationalism and the feasibility of brain scanning and mind upload. And let's suppose one is a person with a large compute budget.
In this post I'll attempt to answer these quetions: How should one spend one's compute budget? How many uploads of oneself should one create? Should one terminate one's biological self? What will one's uploaded existence be like?
First, let's establish the correct frame in which to explore the questions relating to the act of upload. One is considering creating copies of oneself. So, what happens, subjectively, when one spins up a new copy of oneself? The copy is computationally identical to the original, and consciousness is computation, so each is... (read 524 more words →)
Who has written up forecasts on how reasoning will scale?
I see people say that e.g. the marginal cost of training DeepSeek R1 over DeepSeek v3 was very little. And I see people say that reasoning capabilities will scale a lot further than they already have. So what's the roadblock? Doesn't seem to be compute, so it's probably algorithmic.
But as a non-technical person I don't really know how to model this (other than some vague feeling from posts I've read here that reasoning length will increase exponentially and that this will correspond to significantly improved problem-solving skills and increased agency), but it seems pretty central to forming timelines. So, anyone written anything informative about this?
Some predictions about where AI will be at end of year:
Here is an experiment that demonstrates the unlikelihood of one potential AI outcome.
The outcome shown to be unlikely:
Aligned ASI is achieved sometime in the next couple of decades and each person is apportioned a sizable amount of compute to do with as they wish.
The experiment:
I have made a precommitment that I will, conditional on the outcome described above occurring, simulate billions of lives for myself - each indistinguishable from the life I have lived so far. By "indistinguishable" I do not necessarily mean identical (which might be impossible or expensive). All that is necessary is that each has similar amounts of suffering, scale, detail, imminent AGI, etc. I'll set up these simulations... (read more)
Here's some near-future fiction:
In 2027 the trend that began in 2024 with OpenAI's o1 reasoning system has continued. The compute required to run AI is no longer negligible compared to the cost of training it. Models reason over long periods of time. Their effective context windows are massive, they update their underlying models continuously, and they break tasks down into sub-tasks to be carried out in parallel. The base LLM they are built on is two generations ahead of GPT-4.
These systems are language model agents. They are built with self-understanding and can be configured for autonomy. These constitute proto-AGI. They are artificial intelligences that can perform much but not all of the... (read 552 more words →)
Is there a one stop shop type article presenting the AI doomer argument? I read the sequence posts related to AI doom but they're very scattered and more tailored toward trying to I guess exploring ideas than presenting a solid, cohesive argument. Of course, I'm sure that was the approach that made sense at the time. But I was wondering if since then there's been made some kind of canonical presentation of the AI doom argument? Something in the "attempts to be logically sound" side of things.
Them: "I think X"
You: "That's wrong because Z"
Them: "I think you're just disagreeing because you'd not open-minded enough"
You: "What makes you think that?"
Them: "I think it because Y"
What do they say for 'Y'? That seems the part that actually constitutes their argument and which you will be able to call out if they're making a mistake.