Projects to improve AI Safety community epistemic infrastructure seem like high value to me.
The alignment problem seems pretty hard still, but it would feel one to several orders of magnitude harder without LW and all the epistemic infrastructure built up here. I'm talking about both the explicit infrastructure (wikitags, two-axis voting, recommendation algorithms), and implicit (norms and culture). The effort that the moderation team puts in is obviously a huge pillar too. By a naive induction it seems like other bets along this vector would be useful, whether large scope or small scope. Particularly weird/untested ideas. Could be complementary or orthogonal to existing infrastructure.
I've heard that there's about to be a large amount of money coming into AI Safety soon because of the Anthropic IPO. I'm pretty worried about that money being primarily dumped into highly correlated bets like 1,001 more LLM Safety projects. Maybe this could be a good way to diversify?
I don't think its a foregone conclusion that timelines are too short for efforts in this vein to be useful (pick one of: LLMs don't scale to AGI, we get a pause, LLMs scale to AGI but only slowly). And there are maybe even some worlds in which timelines get delayed (Idk, maybe we can better coordinate around a pause, or get one in the first place, or maintain one better once we get it)?
What's this being bottlenecked on right now? Talent or money? Or is there not actually much room left for improvement over what we have right now?
If AI continues to improve, AI-safety is going to go mainstream and the AI-safety community will become eventually 2-4 orders of magnitude larger. If possible you'd want to keep good epistemic norms in that world. The bay area AI-safety community although still influential will become a very small part of the overall safety community. Seems high value to me how to think about keeping good epistemic norms in this world.
Disagreements about alignment difficulty and related issues, such as the feasibility of automating alignment research, are often downstream of disagreements about takeoff speed. With slow takeoff, it's not immediately unreasonable to argue that a pragmatic engineering approach to alignment could work, or that controlled and/or moderately aligned models could help with alignment research. On the other hand, I'm not aware of any remotely ok arguments that fast takeoff wouldn't be disastrous. (Of course, there are unrelated plausible reasons for optimism, such as long timelines or the potential for ending the acute-risk period without building AGI.)
Going further, takeoff disagreements seem partially downstream of disagreements about whether scaling+tweaking LLMs will get us to AGI. If I knew LLMs wouldn't scale to AGI, I'd put a lot less probability on a slow takeoff, and I think so would many others. Unfortunately, it seems to me that resolving the LLM scaling question in the positive is hard because, although absence of evidence is evidence of absence, it's hard to know when you've been exhaustive enough in showing that there aren't any blockers between here and AGI. We don't understand intelligence very well and so there are plenty of blind spots where we could miss a blocker. And resolving the question in the negative requires unearthing and clarifying blockers in a way that is infohazardous. A great way to advance capabilities is to tell capability researchers exactly why their method won't work in a very precise manner. I suspect that this is part of the reason why bearish folks like Thane, Steve, and Tsvi can sometimes be a little vague when describing their beliefs, but you'd have to ask them.
The correlation in beliefs isn't 1-1-1, however. Arguments do exist for why alignment is hard despite slow takeoff, or why slow takeoff is likely even if nothing like LLMs can reach AGI, or why LLM fast takeoff will happen despite what we've seen from LLMs so far. The task is certainly more difficult than if you could definitively answer the LLM scaling question, but it seems possible for progress in strategic clarity to happen by honing in on arguments that hold regardless of the answer.
Alternatively, maybe a convincing non-infohazardous argument against LLMs scaling to AGI could be made. I would love to see Tsvi's argument against the "no blocker intuition" developed and analyzed further, for example.
Or we could just wait and see, but this problem seems too important not to try to make progress on it.
Contrary to the standard advice to work on one habit or self-improvement vector over time, most habits have synergies and interaction effects with other habits and should be adopted in batches. Synergies could include: reminding you of similar habits, making another habit easier to do, giving you more repetitions at another habit per unit of time, or directly working the same thought patterns in a way that applies to multiple habits.
As a personal example, I'm currently focusing on getting more work done by sticking to my schedule more, as well as working with better pace and more playfulness. The former is hard to do without the latter (work is less enjoyable and I want to break my schedule more), and the latter gets more reps when the former is included. I've tried adopting these habits individually and it was much more difficult to stick to them than at present.
Edit 9/29/26: this belief turned out to be quite accurate, but it no longer seems to imply short timelines to me.
When you improve the capabilities of an LLM (or any AI agent with enough generality really) you also probably improve its ability to cooperate with copies of itself, where cooperation ability is some loose collection of abilities such as checking other copies work, planning work for other copies to carry out, course correcting other copies when they get sidetracked, integrating a copy’s work into the main project, etc.
Now that LLMs are good enough to create useful multi-agent systems such as Claude Code, Cursor, Gas Town, etc., I expect these systems to get better at a faster rate than the underlying models (and not because the people are putting work into improving the structure of these systems, although I suspect that will also help).
I’m not sure if these systems are currently more or less than the sum of their parts (if Claude is able to do a task with X difficulty, are 10 Claude agents able to do a task of >10X difficulty, 10X difficulty, or <10X difficulty?). But I expect that as cooperation gains accumulate, they will become less and less like “less than the sum of their parts” (as they stop bungling communication with each other) and more and more like “greater than the sum of their parts” (as they learn how to get gains from specialization or something?).
This has updated me significantly towards shorter timelines, especially for time to automated coders. I really have no idea what happens after that though.
Prediction: 80% chance of fully automated coding “teams” by the end of 2026.