Prosaic Alignment is currently more important to work on than Agent Foundations work.

LLMs as currently trained run ~0 risk of catastrophic instrumental convergence even if scaled up with 1000x more compute

Academia is sufficiently dysfunctional that if you want to make a great scientific discovery you should basically do it outside of academia.

Pursuing plans that cognitively enhance humans while delaying AGI should be our top strategy for avoiding AGI risk

Current progress in AI governance will translate with greater than 50% probability into more than a 2 year counterfactual delay of dangerous AI systems

It is very unlikely AI causes an existential catastrophe (Bostrom or Ord definition) but doesn't result in human extinction. (That is, non-extinction AI x-risk scenarios are unlikely)

Ambitious mechanistic interpretability is quite unlikely^[1] to be able to confidently assess^[2] whether AIs^[3] are deceptively aligned (or otherwise have dangerous propensities) in the next 10 years.

Things will basically be fine regarding job loss and unemployment due to AI in the next several years and those worries are overstated 

The current AI x-risk grantmaking ecosystem is bad and could be improved substantially.

It is critically important for US/EU companies to build AGI before Chinese companies.

People aged 12 to 18 should basically be treated like adults rather than basically treated like children.

EAs and rationalists should strongly consider having lots more children than they currently are

Meaningness's "Geeks Mops and Sociopaths" model is an accurate model of the dynamics that underlie most social movements

Irrefutable evidence of extraterrestrial life would be a good thing.

Someone in the AI safety community (e.g. Yud, Critch, Salamon, you) can currently, within 6 month's effort, write a 20,000 word document that would pass a threshold for a coordination takeoff on Earth, given that 1 million smart Americans and Europeans would read all of it and intended to try out much of the advice (i.e. the doc succeeds given 1m serious reads, it doesn't need to cause 1m serious reads). Copy-pasting already-written documents/posts would count.

It was a mistake to increase salaries in the broader EA/Rationality/AI-Alignment ecosystem between 2019 and 2022

Good AGI-notkilleveryoneism-conscious researchers should in general prioritize working at big AGI labs over working independently, for alignment-focused labs, or for academia marginally more than they currently do.

The ratio of good alignment work done at labs vs independently mostly skews toward labs

Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than independent researchers. Your answer should factor out such considerations.

Edit: Also normalize for quantity of researchers.

Empirical agent foundations is currently a good idea for a research direction.

There is a greater than 20% chance that the Effective Altruism movement has been net negative for the world.

A basic deontological and straightforward morality (such as that exmplified by Hermione in HPMOR) is basically right; this is in contrast with counterintuitive moralities that suggest evil-tinted people (like Quirrell in HPMOR) are also valid ways of being moral.

The work of agency-adjacent research communities such as artificial life, complexity science and active inference is at least as relevant to AI alignment as LessWrong-style agent foundations research is.

Just like the last 12 months was the time of the chatbots, the next 12 months will be the time of agent-like AI product releases.

American intelligence agencies consider AI safety to be substantially more worth watching than most social movements

It is possible to make meaningful progress on deceptive alignment using experiments on current models

Moloch is winning.

"Polyamory-as-a-default-option" would be a better social standard than "Monogamy-as-a-default-option".

The rationality community will noticeably spill over into other parts of society in the next ten years. Examples: entertainment, politics, media, art, sports, education etc.

Among existing alignment research agendas/projects, Superalignment has the highest expected value

At least one American intelligence agency is concerned about the AI safety movement potentially decelerating the American AI industry, against the administration/natsec community's wishes

Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large compute budgets for safety research teams.

I broadly agree with the claim that "most people don't do anything and the world is very boring".

On the current margin most people would be better off involving more text-based communication in their lives than in-person communication.

Language model agents are likely (>20%) to produce AGI (including the generalization to foundation model-based cognitive architectures)

Agent foundations research should become more academic on the margin (for example by increasing the paper to blogpost ratio, and by putting more effort into relating new work to existing literature).

Current AI safety university groups are overall a good idea and helpful, in expectation, for reducing AI existential risk 

Current progress in AI governance will translate with greater than 20% probability into more than a 2 year counterfactual delay of dangerous AI systems

Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large-scale lobbying efforts in DC.

Immersion into phenomena is better for understanding them than trying to think through at the gears-level, on the margin for most people who read LessWrong.

Rationality should be practiced for Rationality’s sake (rather than for the sake of x-risk).

It is possible to make meaningful progress on ELK using empirical experiments on current models

Public mechanistic interpretability research is net positive in expectation.

Rationalists would be better off if they were more spiritual/religious

Effective altruism can be well modeled by cynically thinking of it as just another social movement, in the sense that those a part of it are mainly jockeying for in-group status, and making costly demonstrations to their in-group & friends that they care about other sentiences more than others in the in-group. Its just that EA has more cerebral standards than others.

Most LWers should rely less on norms of their own (or the LW community's) design, and instead defer to regular societal norms more.

"Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world" is a quite probable point-of-no-return regarding AGI risk.

Investing in early-stage AGI companies helps with reducing x-risk (via mission hedging, having board seats, shareholder activism)

"Agent" is an incoherent concept.

The younger generation of rationalists are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.

The most valuable new people joining AI safety will usually take ~1-3 years of effort to begin to be adequately sorted and acknowledged for their worth, unless they are unusually good at self-promotion e.g. gift of gab, networking experience, and stellar resume.

Poll feature on LW: Yay or Nay?

At least one of {Anthropic, OpenAI, Deepmind} is net-positive compared to the counterfactual where just before founding the company, its founders were all discretely paid $10B by a time-travelling PauseAI activist not to found the company and to exit the industry for 30 years, and this worked.

One should basically not invest into having "charisma".

Language model agents are very likely (>80%) to produce AGI (including the generalization to foundation model-based cognitive architectures)

Great art is rarely original and mostly copied.

There is a greater than 80% chance that effective altruism has been net-negative for the world.

If rationality took off in China, it would yield higher EV from potentially spreading to the rest of the world than from potentially accelerating China.

"Intelligence" can be characterized with a similar level of theoretical precision as e.g., heat, motion, and information. (In other words: it's less like a messy, ad-hoc phenomena and more like a deep, general fact about our world). 

When people try to discuss philosophy, math, or science, especially pre-paradigmatic fields such as ai safety, they use a lot of metaphorical thinking to extend from familiar concepts to new concepts. It would be very helpful and people would stop talking past each other so much if they practiced being explicitly aware of these mental representations and directly shared them rather than pretending that something more rigorous is happening. This is part of Alfred Korzybski's original rationality project, something he called 'consciousness of abstraction.'

Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.

A Secular Solstice variation designed to work weekly (akin to Sunday Service or Shabbat) would be positive for rationalists, both for community and for the thought processes of the members.

Most persistent disagreements can more usefully be thought of as a difference in priors rather than a difference in evidence or rationality.