tldr. some unrefined / semi-stream-of-consciousness personal anecdotes, thoughts on the METR report, and what we can do in the few months/year (?) left till superhuman intelligence
stars
I cried for the first time in several months yesterday. I still don't know if I'm crazy, but the person I was just half a year ago certainly would have thought I was.
Eight months ago, I was dead set on becoming an astrophysicist. I had just clicked the "view decision" button on my college admissions portal, screeching inside with joy when my screen had erupted in confetti. The hundreds of words I'd written professing my love for the dark had paid off; I was going to be at a place where discovery was all they did. I was going to discover what made up dark matter, I was going to discover the force behind dark energy, I was going to discover the new . I was going to be a scientist! Two months after opening the letter, I had a RA position lined up at my university's Center for Astrophysics, where I would be on a team discovering the most distant galaxies in the universe. For my entire life, I had conceptualized myself as having one primary goal: to discover new and interesting pieces of knowledge. And what could be more interesting than probing the edge of the universe?
Five months ago, I came across something called mechanistic-interpretability. I liked how the words beat staccato against my mouth. I liked how smart I sounded when I told someone I was interested in it, though the actual reason for my interest was fairly more pragmatic: I wanted to understand how the models I was building to discover the faintest objects in the sky actually worked.
Four months ago, I stumbled across LessWrong. And Astral Codex Ten. And Neel Nanda. It is difficult to re-imagine my headspace all those months ago, since that world is so detached from the one I live in today, but if I had to describe it, I would say:
"Wow, look at all these nerds thinking about all of these super interesting things! I don't fully buy these AI takeoff scenarios, but I like how they reason; this 'rationalist' thing seems pretty aligned with my worldview."
Halfway into the month, Hugging Face was disclosed to the public. A lot of things have transpired since Hugging Face. I've been pulled into the pipeline of AI safety courses and research fellowships. If I had to describe my attitude a month ago, I would say:
"My p(doom) is probably on the order of 5%. I believe that AI is likely the highest source of p(doom) we have currently, but I believe in bottlenecks + the indomitable will of humanity to make things go relatively well."
My p(doom) is now on the order of 40%. Yesterday, I cried about the METR report (if you somehow haven't read it, the one about research acceleration inside Anthropic). I cannot remember the last time I've genuinely thought about being an astrophysicist.
Timelines have always been a tricky concept to me; every time someone asked me about my own, I would immediately defer to "I have very large errorbars." This is not to say that I was completely ignorant of timelines: I too had read AI 2027 and spent a week processing what the hell I had just read. But the lizard brain in my head, so stuck in its ways of inertia, would always defer to "they're just being overzealous." [1]
On a further point, my timelines were fuzzy due to a large dose of motivated reasoning. When you do perturbation theory in physics, you often discard terms of higher order because they contribute negligible amounts to your final result. When I think about the potential impact I may have on the world, short timelines tend not to contribute at all to the final calculation; after all, if ASI arrives in 1 month, there is little I can do to make the world update in a safer direction. I now recognize that it is this impact-motivated reasoning, which I suspect a lot of people who have not converted to short timelines have adopted, that placed the proverbial blinders around my head.
I am still figuring out why the METR report has provoked such a response in me. Perhaps it is Nikola Jurkovic's time horizons conversion that indicates we will reach tasks on the order of years by 2027 (whatever that means). Perhaps it is the AI Futures Project analysis that we are 5 months ahead of AI 2027. Perhaps it is, despite all my cope that "oh, capabilities are constrained by pre-training, or RL generalization will never happen, or fuzzy tasks are difficult, or even that logistic growth often looks exponential until you near the plateau," METR's unabashed statement that "We did not see evidence for significant bottlenecks[2]."
It seems increasingly likely that ASI will arrive in the next couple of years. If we cannot figure out superalignment by then, which seems like an increasingly likely possibility, we will have RSI with a misaligned intelligence, in which case Eliezer can run down the whole list of arguments of why that leads to human extinction. In the case we get an aligned ASI, I do not know what to do in such a world. I have optimized my entire life for intellectual labor, to look out at the stars and discover cool shit. What happens when a machine will be able to do all of that and more?
Thus, I have figured out the reason I was crying: this is not the world I wanted to grow up in. I have always thought that I wanted to live in a millenarian world. My childhood was full of books describing galactic civilizations and transhuman utopias. I was always excited about each election as the tipping point, morbidly fascinated with any world conflicts that seemed to be breaking out, and somewhat strangely, excited about COVID? In a sense, I was a naive accelerationist: excited for the old world to wash away into the new. Somewhat selfishly, now that this millenarian tipping point seems inevitable, I feel a strong tendency in the opposite direction, to preserve the status quo.
I have grown comfortable in this world, and I admit I have implicitly lived my life to one day grow up in it. I have dreamed of being great, of having my name in the textbook of some interplanetary civilization 500 years in the future. I have dreamed of living a happy life with a person I love more than anyone on this planet. I have dreamed of taking life-changing roadtrips across continents with my best friends. I have dreamed of being a humanitarian, of helping millions across the planet live their lives better. Of course, these are grand ambitions, and some may even be mutually exclusive. But by the time I turn 19, the world that these ambitions live in may not exist.
There is a current flowing through this world, and I feel unable to swim against it. I want to go back to dreaming about stars.
blinders
It is difficult to reconcile these emotions with the outside world. Perhaps this melancholy is a symptom of "lack-of-grass-touching-itis." Perhaps it is a symptom of high screen times on Twitter and Chrome (LessWrong). I try to ask myself why everyone does not care about this, why everything feels so ordinary, why people still talk about petty politics and celebrity gossip and he-said-she-said. I ask myself why our AI regulation protests are still small and why only 2 of the people in the introductory AI safety fellowship I teach had heard of Hugging Face before taking the class. I ask myself why we still teach introductory AI safety fellowships.
I think the answer is that I was once one of these people: so adamant in the world I wanted to live in that I willingly chose to put blinders on myself. It is by accident that I stumbled upon the signs of this great wave moving towards the lightcone. Even then, for a month or two, I had convinced myself of long timelines. I was still diligently completing my p-sets, GPA-optimizing, harboring crushes, planning for grad school. Everyone still seems to have the same blinders on.
I feel like Cassandra, given the gift of prophecy but cursed such that no one will believe me. [3] I sound like a delusional maniac with a bit too much blue-light exposure when I tell my friends about AGI and ASI and RSI and SAR and the outmoding of all human intellectual labor, when I tell them about misalignment and behavioral selection models, and influence seeking, and how these models are optimized to deceive us. Who would believe that their futures and lives and desires will be whisked away in a year? That billions of people on this planet could die, or that we could be reduced to living out simulations in the pleasure machines of some Bostromian Deep "Utopia."
But spreading the word is all that is left. I am incredibly bearish that alignment, much less superalignment, will be solved by mid-end 2027. Thus, there seems no other choice but to scream. We must scream at the labs, the politicians, the oligarchs, and most of all (unfortunately), the people. We must rip their blinders off before we are washed away. We must unilaterally pause and figure out what the fuck is going on. This does not only mean solving alignment, but defining what kind of world we may want to live in post-ASI. This may sound strange (and likely is an outlier opinion), but I am not sure I want to live in a post-scarcity, post-knowledge "utopia."
We must tell everyone to look around.
tsunami
When the 2004 Indian Ocean Tsunami happened, the victims were sunbathing and sipping margaritas on the beach. When the water started receding, rather than running for the hills, more people came to the shore to see the now-exposed reefs and mud flats. Minutes later, waves dozens of feet high crashed into the shore, eventually killing 230,000 people across the region.
The water is receding. The S&P continues to boast new highs. We are witnessing new technologies and capabilities that would seem like magic even a few decades ago. The lives of the average San Franciscan have never been better. We are scrambling towards the shoreline, founding AI B2B SaaS startups and making expert RL environments. We are rolling in the sands of intelligence, awash in number-go-up-isms.
There are some screams to be heard amongst the crowd. You can hear some people running in the other direction, because they recognize what the water receding truly means. Please look at the horizon. Please feel the rumbling beneath you.
I would say this thinking is somewhat derivative of the mental correction I made after the 2016/2024 elections to not be over-zealous about any particular outcome.
tldr. some unrefined / semi-stream-of-consciousness personal anecdotes, thoughts on the METR report, and what we can do in the few months/year (?) left till superhuman intelligence
stars
I cried for the first time in several months yesterday. I still don't know if I'm crazy, but the person I was just half a year ago certainly would have thought I was.
Eight months ago, I was dead set on becoming an astrophysicist. I had just clicked the "view decision" button on my college admissions portal, screeching inside with joy when my screen had erupted in confetti. The hundreds of words I'd written professing my love for the dark had paid off; I was going to be at a place where discovery was all they did. I was going to discover what made up dark matter, I was going to discover the force behind dark energy, I was going to discover the new
Five months ago, I came across something called mechanistic-interpretability. I liked how the words beat staccato against my mouth. I liked how smart I sounded when I told someone I was interested in it, though the actual reason for my interest was fairly more pragmatic: I wanted to understand how the models I was building to discover the faintest objects in the sky actually worked.
Four months ago, I stumbled across LessWrong. And Astral Codex Ten. And Neel Nanda. It is difficult to re-imagine my headspace all those months ago, since that world is so detached from the one I live in today, but if I had to describe it, I would say:
Halfway into the month, Hugging Face was disclosed to the public. A lot of things have transpired since Hugging Face. I've been pulled into the pipeline of AI safety courses and research fellowships. If I had to describe my attitude a month ago, I would say:
My p(doom) is now on the order of 40%. Yesterday, I cried about the METR report (if you somehow haven't read it, the one about research acceleration inside Anthropic). I cannot remember the last time I've genuinely thought about being an astrophysicist.
Timelines have always been a tricky concept to me; every time someone asked me about my own, I would immediately defer to "I have very large errorbars." This is not to say that I was completely ignorant of timelines: I too had read AI 2027 and spent a week processing what the hell I had just read. But the lizard brain in my head, so stuck in its ways of inertia, would always defer to "they're just being overzealous." [1]
On a further point, my timelines were fuzzy due to a large dose of motivated reasoning. When you do perturbation theory in physics, you often discard terms of higher order because they contribute negligible amounts to your final result. When I think about the potential impact I may have on the world, short timelines tend not to contribute at all to the final calculation; after all, if ASI arrives in 1 month, there is little I can do to make the world update in a safer direction. I now recognize that it is this impact-motivated reasoning, which I suspect a lot of people who have not converted to short timelines have adopted, that placed the proverbial blinders around my head.
I am still figuring out why the METR report has provoked such a response in me. Perhaps it is Nikola Jurkovic's time horizons conversion that indicates we will reach tasks on the order of years by 2027 (whatever that means). Perhaps it is the AI Futures Project analysis that we are 5 months ahead of AI 2027. Perhaps it is, despite all my cope that "oh, capabilities are constrained by pre-training, or RL generalization will never happen, or fuzzy tasks are difficult, or even that logistic growth often looks exponential until you near the plateau," METR's unabashed statement that "We did not see evidence for significant bottlenecks[2]."
It seems increasingly likely that ASI will arrive in the next couple of years. If we cannot figure out superalignment by then, which seems like an increasingly likely possibility, we will have RSI with a misaligned intelligence, in which case Eliezer can run down the whole list of arguments of why that leads to human extinction. In the case we get an aligned ASI, I do not know what to do in such a world. I have optimized my entire life for intellectual labor, to look out at the stars and discover cool shit. What happens when a machine will be able to do all of that and more?
Thus, I have figured out the reason I was crying: this is not the world I wanted to grow up in. I have always thought that I wanted to live in a millenarian world. My childhood was full of books describing galactic civilizations and transhuman utopias. I was always excited about each election as the tipping point, morbidly fascinated with any world conflicts that seemed to be breaking out, and somewhat strangely, excited about COVID? In a sense, I was a naive accelerationist: excited for the old world to wash away into the new. Somewhat selfishly, now that this millenarian tipping point seems inevitable, I feel a strong tendency in the opposite direction, to preserve the status quo.
I have grown comfortable in this world, and I admit I have implicitly lived my life to one day grow up in it. I have dreamed of being great, of having my name in the textbook of some interplanetary civilization 500 years in the future. I have dreamed of living a happy life with a person I love more than anyone on this planet. I have dreamed of taking life-changing roadtrips across continents with my best friends. I have dreamed of being a humanitarian, of helping millions across the planet live their lives better. Of course, these are grand ambitions, and some may even be mutually exclusive. But by the time I turn 19, the world that these ambitions live in may not exist.
There is a current flowing through this world, and I feel unable to swim against it. I want to go back to dreaming about stars.
blinders
It is difficult to reconcile these emotions with the outside world. Perhaps this melancholy is a symptom of "lack-of-grass-touching-itis." Perhaps it is a symptom of high screen times on Twitter and Chrome (LessWrong). I try to ask myself why everyone does not care about this, why everything feels so ordinary, why people still talk about petty politics and celebrity gossip and he-said-she-said. I ask myself why our AI regulation protests are still small and why only 2 of the people in the introductory AI safety fellowship I teach had heard of Hugging Face before taking the class. I ask myself why we still teach introductory AI safety fellowships.
I think the answer is that I was once one of these people: so adamant in the world I wanted to live in that I willingly chose to put blinders on myself. It is by accident that I stumbled upon the signs of this great wave moving towards the lightcone. Even then, for a month or two, I had convinced myself of long timelines. I was still diligently completing my p-sets, GPA-optimizing, harboring crushes, planning for grad school. Everyone still seems to have the same blinders on.
I feel like Cassandra, given the gift of prophecy but cursed such that no one will believe me. [3] I sound like a delusional maniac with a bit too much blue-light exposure when I tell my friends about AGI and ASI and RSI and SAR and the outmoding of all human intellectual labor, when I tell them about misalignment and behavioral selection models, and influence seeking, and how these models are optimized to deceive us. Who would believe that their futures and lives and desires will be whisked away in a year? That billions of people on this planet could die, or that we could be reduced to living out simulations in the pleasure machines of some Bostromian Deep "Utopia."
But spreading the word is all that is left. I am incredibly bearish that alignment, much less superalignment, will be solved by mid-end 2027. Thus, there seems no other choice but to scream. We must scream at the labs, the politicians, the oligarchs, and most of all (unfortunately), the people. We must rip their blinders off before we are washed away. We must unilaterally pause and figure out what the fuck is going on. This does not only mean solving alignment, but defining what kind of world we may want to live in post-ASI. This may sound strange (and likely is an outlier opinion), but I am not sure I want to live in a post-scarcity, post-knowledge "utopia."
We must tell everyone to look around.
tsunami
When the 2004 Indian Ocean Tsunami happened, the victims were sunbathing and sipping margaritas on the beach. When the water started receding, rather than running for the hills, more people came to the shore to see the now-exposed reefs and mud flats. Minutes later, waves dozens of feet high crashed into the shore, eventually killing 230,000 people across the region.
The water is receding. The S&P continues to boast new highs. We are witnessing new technologies and capabilities that would seem like magic even a few decades ago. The lives of the average San Franciscan have never been better. We are scrambling towards the shoreline, founding AI B2B SaaS startups and making expert RL environments. We are rolling in the sands of intelligence, awash in number-go-up-isms.
There are some screams to be heard amongst the crowd. You can hear some people running in the other direction, because they recognize what the water receding truly means. Please look at the horizon. Please feel the rumbling beneath you.
Please open your eyes.
I would say this thinking is somewhat derivative of the mental correction I made after the 2016/2024 elections to not be over-zealous about any particular outcome.
I sincerely hope this ages poorly or is a function of the redaction by Anthropic.
I wonder if this is how Eliezer feels.