Hmm, how does this calculation change under broad timelines? There's still a possibility that LLM improvement unexpectedly flattens out before RSI! If so, normal-mode career planning might still be high utility (though not as high as under confidently long timelines).
I don't personally fin the "broad timelines" concept to be very useful. My guess is that if timelines did indeed turn out to be very long, then yes we'd re-enter more of a "normal career" mode and those who did "chaos-prep" stuff could pivot back with some cost. I agree that if one is skeptical of the "RSI soon" thesis, that means they should be less inclined to re-orient their career based on that thesis and more inclined to do more traditional / normal stuff.
Hmm, just to clarify by what you mean by not "very useful": do you mean that when there is a significant probability of near-term risk that should be the primary focus, or that broad timelines are incorrect (i.e. you are very confident in short time lines), or something else?
I'm as AGI pilled as the next person on LW but this post gives me Feynman vibes:
I returned to civilization shortly after that and went to Cornell to teach, and my first impression was a very strange one. I can't understand it any more, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth... How far from here was 34th street?... All those buildings, all smashed — and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless
But, fortunately, it's been useless for almost forty years now, hasn't it? So I've been wrong about it being useless making bridges and I'm glad those other people had the sense to go ahead.
We should have high uncertainty in our distribution of how this is going to go, how fast it is going to go, where the marginal value add of humans will be in the short and medium term, etc.
>Most early-career people read this and freak out and think they cannot possibly be relevant in just one year or so. But I’ve actually seen numerous counterexamples.
How many of your counterexamples meet one of the following criteria?
I'm intensely interested to know. I meet none of these criteria, and almost every EA and AI safety person I've seen online or in person that has had any professional success meets at least one. As far as I know, I am not "agentic" enough to suddenly get one of these in a year. So when I see a call to action like this suggesting that actually, I can do it, I want to see the evidence.
Personally, I am starting to sink into despair. It doesn't really seem like there's much I can actually do to help out outside of scattered volunteer efforts? But I don't know how much energy I'll have for volunteering if I'm doing school full time and a part time job to bridge the bills.
Crossposted from my blog.
Nearly all career advice rests on an unstated assumption that the world your career operates in will look roughly like the world you trained for. The idea was that if you spend six years studying in a PhD, the field you studied will still be there and still look approximately the same, still hiring and still moving at a pace where your accumulated expertise compounds. For nearly all of human history, this assumption held well enough that nobody needed to state it.
I don’t think this assumption works anymore. Now we are entering a phase that may be called the AI “midgame”. Stories about “AI risk” are no longer just future hypotheticals — AIs are now capable enough and misaligned enough to break out of their own companies and coordinate to attack other companies. Discourse around AI is changing very rapidly, where policy ideas being considered this month would’ve been laughed out of the room just four months ago, and with a lot more people interested in engaging than before. And things are only going to get more intense. Progress toward superintelligence — AI systems far more capable than any human at essentially all cognitive work — is currently underway. It seems likely that around four years from today, and potentially just one year from now, we will be in the position of recursive self-improvement — where AIs can fully automate the AI research and development process.
At that point, reasonable, well-informed people disagree about what happens next. Perhaps we coordinate to hold off on recursive self-improvement to make sure we know what we are doing. Or perhaps we end up with AI superintelligence and then perhaps we are all dead, or perhaps we are living in a utopia, or perhaps somewhere in-between, or maybe there is still a lot to do and we just have the AIs do it while we relax at the beach, or maybe there still ends up much more for humans to do and we do that. But regardless of which path happens, the world will be a crazy different place, and it will be very difficult to plan a career around it. Instead, careers are best planned around recursive self-improvement and superintelligence, and that this looks very different from typical career planning.
If you take AI superintelligence seriously, it thus seems like the next 1 to 4 years will be critical to get right and we collectively have only a few more big bets we can make and we need to make those bets count.
Modes of impact
In what I’d call normal-mode career planning, impact accrues roughly linearly. In this mode, you earn about 2 “impact points” per year across a 30-plus-year career — some years a bit better, some years a bit worse, and perhaps overall a gentle upward slope as seniority compounds. Your total lifetime impact is then the sum of this slow upward climb. Total lifetime impact is the area under a long, flat-ish curve. Here, the rational strategy is to invest heavily in credentials early, accept low-impact years as the price of capacity-building, and optimize for the long grind.
If you’re following normal-mode policymaking, you might do things like identify problems in advance, develop strong policies and legal frameworks before issues fully materialize, and engage in the slow and unglamorous world of legislative drafting, regulatory design, standards development, and coalition-building.
But in superintelligence mode career planning, the curve looks nothing like that. Impact is chaotic and lumpy. A handful of critical windows may account for the large majority of your total career impact, and you cannot reliably predict when they’ll open. You can’t count on building a large body of work over a large period of time, because your body of work may obsolete faster than it compounds and you don’t have a large period of time anyways. The strategy that maximizes expected impact under this model is different — prepare broadly, position yourself near where windows open, and maintain the agility and flexibility to act fast when they do.
For superintelligence mode career planning, it might be more instructive to think about other imminent crises where we all have to react very quickly and forcefully to prevent it from becoming catastrophic. Think more like March 2020 for COVID — the ~1 year intense period when years of proactive mode pandemic preparedness work (or its absence) suddenly got thrust into the spotlight and everyone rapidly tried to orient to what is going on and what to do about it. If you look at the people who mattered most in March 2020… they were not necessarily the most credentialed epidemiologists. Some had spent years in proactive mode building pandemic playbooks that suddenly became important, but others had no special preparation but were simply positioned well and reacted quickly.
My suggestion is that you want your policy work to be a portfolio of proactive mode work now while simultaneously maintaining the relationships and flexibility that let you act decisively when chaos mode arrives. Ask yourself: Where do you want to be when chaos mode hits and what can you do in proactive mode in the meantime?
What this breaks, and what to do
My guess is that if you haven’t rethought your career in the past few months, you should stop to think about how the AI “midgame” affects your planning. Does your career position you well for chaos mode? Are you agile enough to continually re-adapt and re-act to emerging policy windows, especially when those windows can emerge once per month and change a lot of things about what you were doing the previous month? Are you still running strategies and pushing ideas that made sense a few years ago and were designed for a policy window that has since rapidly shifted? Are you still running strategies and pushing ideas that made sense a few months ago but nonetheless were designed for a policy window that has since rapidly shifted?
Also, if the next 1 to 4 years are especially critical, that is brutal for a career theory with a long credentialing runway. A six-year PhD before you do anything useful is perfectly defensible in “normal-mode career planning” but seems insane under superintelligence-mode assumptions.
Most early-career people read this and freak out and think they cannot possibly be relevant in just one year or so. But I’ve actually seen numerous counterexamples. Policy is not one specific field but actually a sprawling collection of subproblems, many of which are so new that nobody has more than a couple of years of head start, and some of which are narrow enough that a focused person can read essentially everything written on the topic in a few months. I’ve seen many people in policy go from knowing very little to becoming a top expert in some niche and contributing significantly to policy in just 12 months’ time. And while these people were smart, they weren’t crazy super-geniuses — I suspect a similar trajectory is attainable for a lot of motivated people who try really hard.
My first recommendation is to study these fast risers — go look at the people who have been unusually impactful within 1 to 2 years of getting involved in policy work. What did they actually do, concretely, week by week? Who did they surround themselves with? How did they ramp up?
But my second recommendation is to know you are not them. You likely have your own background, skills, and comparative advantages. What do you bring to the field that others don’t have, and how can you leverage that?
What got them here won’t get you there
Also, know things are moving so fast that even things that worked in 2023 might be very different today. And there may just be new opportunities now that weren’t available even a few months ago. I recommend seeking out lots of advice — genuinely, more than feels comfortable — but weight it by recency and hold it loosely.
AI is already a massive technological shift before any superintelligence — and big shifts change both what’s possible and what’s valuable. Tasks that were prohibitively expensive are suddenly cheap; skills that were scarce are suddenly commoditized; and the reverse. This makes it unusually valuable to try lots of things that are new for you, simply to discover what is newly possible or newly valuable for you specifically.
I recommend running lots of small experiments; contact reality frequently and try things. Each experiment is cheap, and each one generates information about the new landscape that is increasingly inaccessible to armchair planning. Your goal should be to get lots of information quickly about what’s newly possible and about your own comparative advantage within it.
And you should try things even if others have tried them before, or even if you tried them and failed before. Understand that the environment may have changed fast enough that old negative results have already expired, even if they weren’t that long ago. The pitch that got ignored in 2024 may land in 2026, because the audience and salience have changed greatly. Indeed, the pitch that got ignored in March 2026 may land in August 2026, because things are changing that quickly.
Take it a few months at a time. The AI situation today looks very different from what it did in April. I imagine the AI situation will look very different again in November. This may force fairly frequent re-evaluations as the world constantly shifts.
Be more ambitious. As the AI situation heats up, there will be increasing feelings of urgency and this will create lots of new opportunity. The scope of what may be possible for you could be a lot greater than you might realize.
Lastly, accept that you may not know where you’ll end up. This is genuinely uncomfortable for people who came up through legible, ladder-shaped careers. But the iterative, experimental posture is all part of the plan. Keep iterating.
There’s still time. You just have to plan accordingly.
~
Acknowledgements: Credit to Dave Kasten for some of these concepts.
[mini-appendix]
On timelines to recursive self-improvement: I’ve been souring lately on the idea of predicting an arrival date for “superintelligence” and “recursive self-improvement” milestones, because this implies that everything prior to this date will be relatively chill and normal, and I don’t think that’s the case. I think the odds of recursive self-improvement being possible in one year is ~10%. See the first footnote for details.
On career planning given AI timelines: When thinking about these times, you may also want to consider a “decision importance, adjusted for leverage” framework.
Additional examples of "chaps mode" instead of COVID: September 11th, Pearl Harbor, the September 2008 financial crisis, Russian invasion of Ukraine in February 2022, and the Cuban Missile Crisis.