Here be cynical opinions with little data to back them.
It's important to point out that "AI Safety" in an academic context usually means something slightly different from typical LW fare. For starters, as most AI work descended from computer science, its pretty hard [1] to get anything published in a serious AI venue (conference/journal) unless you
Both PhD students and their advisors want to publish things in established venues, so by default one should expect academic AI Safety research to have a near-term prioritization and be less focused on AGI/ex-risk. That isn't to say research can't accomplish both things at once, but its worth noting.
Because AI Safety in the academic sense hasn't traditionally meant safety from AGI ruin, there is a long history of EA aligned people not really being aware of or caring about safety research. Safety has been getting funding for a long time, but it looked less like MIRI and more like the University of York's safe autonomy lab [2] or the DARPA Assured Autonomy program [3]. With these dynamics in mind, I fully expect the majority of new AI safety funding to go to one of the following areas:
The thing that probably won't get funding is aligning a fully autonomous agent with the implicit interests of all humans (not just trainers), which generalizes to the ex-risk problem. Perhaps I lack imagination, but with the way things are I can't really imagine how you get enough published in the usual venues about this to build a dissertation out of it.
[1] Yeah, of course you can get it published, but I think most would agree that its harder to get a pure theory ex-risk paper published in a traditional CS/AI venue than other types of papers. Perhaps this will change as new tracks open up, but I'm not sure.
[2] https://www.york.ac.uk/safe-autonomy/research/assurance/
Hmm again it depends on whether you're defining "alignment" narrowly (the technical problem of getting superhumanly powerful machines to robustly attempt to do what humans actually want) or more broadly (eg the whole scope of navigating the transition from sapiens controlling the world to superhumanly powerful machines controlling the world in a way that helps humans survive and flourish)
If the former, I disagree with you slightly; I think "human values" are possibly broad enough that some recognizably-human values are easier to align than others. Consider... (read more)
Epistemic Status: I had the idea for the post a few days ago and quickly wrote it down while on a train. I'm very curious about other perspectives.
TL;DR: The recent increased public interest in AI Safety will likely lead to more funding for and more researchers from academia. I expect this increase to be larger than that of non-academic AI Safety work. We should prepare for that by thinking about how we "onboard" new researchers and how to marginally allocate resources (time and money) in the future.
Why I think academia's share in AI safety will increase
With the recent public interest in AI (existential) safety, many people will think about how they can help. Among people who think "I might want to do research on AI Safety", most will come from academia because that's where most research happens. Among people who will think "I should fund AI Safety research", most will fund academic-style research because that's where most research talent sits, and because it's the "normal" thing to do. I expect this increase to be larger than that of AI Safety researchers in companies (though with less certainty), AI Safety orgs, or independent researchers of, e.g., the "Lesswrong / Alignment Forum" style.
Weak evidence that this is already happening
At the university of Amsterdam, where I'm a PhD student, there has been increased interest in AI Safety recently. In particular, one faculty actively starts to think about AI existential safety and wants to design a course that will include scalable oversight, and ≥4 other faculty are at least starting to get informed about AI existential safety with an "open mind".
What might one do to prepare?
Needless to say, I didn't think about this a lot, so take the following with a grain of salt and add your own ideas.
Uncertainties
I find it plausible that the representation of AI Safety researchers in companies like OpenAI and DeepMind will also grow very fast, though I think the increase will be smaller than in academia.