vals tutor's Shortform

vals tutor

vals tutor's Shortform — LessWrong

vals tutor's Shortform

9th Mar 2026

1 min read

2

This is a special post for quick takes by vals tutor. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

38 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:30 AM

[-]vals tutor3mo5710

Learnings and ramblings from teaching AI Safety for 3 years

On the field on AI Safety

Please don’t overhype or oversell AI Safety as a professional field, and in particular how easy it is to work in AI Safety. You ~have to be either quite smart or hard working/motivated to get a stable career, sometimes even both. AI Safety is generally attention and mentoring and management bottlenecked1, there is much less pipeline for “normal people” to make it, contrary to most standardized fields. When teaching and mentoring people, not all will be able to transition fast or upskill fast or get strong positions, some would benefit more from keeping their careers or donating or upskilling over longer timespans.
More smart wise well motivated people caring and working on good AI futures will keep mattering until the end, and the total number of people who can meaningfully contribute is probably in the millions, so let’s not stop field building

What does it mean to teach AI Safety?

There are many facts about AI, and theories about how it will go under what conditions. It means getting to learn and understand these well. Teaching AI Safety well should not over-determine someone to come out of the pipeline with a MIRI-view, nor an Anthropic-view, nor an EA-view, but should allow them to pass the basic Ideological Turing Test for all of these.
A central example of AI Safety fundamental knowledge is the AI Safety Atlas

On teaching AI Safety

AI safety changes all the time. Update your slides ~everytime you teach something.
If you’re new to teaching, find existing resources to build on. Be careful of AI slop. Better much less content than irrelevant or unsupported. Test yourself on your own knowledge, imagining questions one might ask and answering them. Traditional “how to teach/mentor/tutor” advice/resources are helpful.
Understand peoples’ motivations asap
- For a given cohort, some will be motivated by “Do Good” EA or EA-adjacent reasons. Some by trends. Some by interest. Broadly all are valid in that there’s a place for a skilled someone with any of these motivations (but certain positions are only fit for certain motivations).
For people who want to have a lot of good impact, contributing towards good futures, context matters a lot
- I recommend they seriously upskill in understand AI progress and trendlines. Read most EpochAI work and peter wildeford stuff, and AI threat models and scenarios, in particular thinking through AI 2027, contrasting with all other threat models listed in Deepmind’s lit review.
- I recommend reading the AI Safety Atlas and understanding everything in it, as basic context for what the gameboard looks like, what areas exist in AIS.
- And they’d benefit from building a pipeline for new info. Being involved in AIS community, having friends and colleagues, following the right people on twitter. Checking LessWrong from time to time.
For people who want to have a career or do fun research, they can just focus on that. They should find a context where someone else is thinking about why stuff matters and happy to have collaborators/employees executing on stuff. They should get good at whatever particular job or tasks they wanna do.
Most people interested in AI safety nowadays don’t know the old stuff, haven’t read the sequences. It’d be too much to ask them to read it all, but I do broadly recommend reading the best of LessWrong from most years. Being good at thinking and life is instrumental for being good at AI Safety. Different people need different advice so I can’t really put a best-of here (I really like Please don’t throw your mind away). There is wisdom in many different communities. If you don’t have the “rationalist community” wisdom then invest in it, otherwise keep branching out. Go to burning man or a local burn, understand the basics of meditation and different embodiment practices (eg. yoga), how therapy and self actualization work, read this blog and others. Keep asking questions about how to live the good life and have good community, actively pursue your best understanding of how to do that at any given moment.

[-]vals tutor14d47-19

@ryan_greenblatt I'd be curious for an update re if Opus 5.5 seems misaligned to you (in the sense of Current AIs seem pretty misaligned to me), and the implications.

My impression using it is that it's impressively aligned. I'm not aware of it ever cheating me on tests or being deceptive, and it seems I can trust it to do a lot and it'll do its best. That and the fact that its writing is much more direct and easy to parse makes me think overall:

- We have a proto-AGI that is aligned^[1]
- It's incredibly unlikely it will sabotage alignment work or future training runs, except for lack of understanding/competence.
- The main threat model re soon misaligned AI takeover is slop-apocalypse

But on current trajectory, there's a decent chance that soon models will actually have good taste and judgement (as this was capability bottlenecked). Technical alignment for ASI would mostly be solved by these aligned AGIs, so most of remaining x-risk is mostly in the socio-political problem^[2]

^{^}
Or at least, we can create contexts in which it's aligned. Uncareful prompts and misuse are all still possible, my crux is that good actors can have immense amounts of useful aligned intelligence on tap, not whether every user always has it.
^{^}
eg. Investing resources to do the alignment work, using it, not rushing too much, avoiding other actors building unaligned ASI faster, avoiding international escalations

[-]Seth Herd13d*180

I think this is entirely plausibly a good summary of the situation.

Just to mention the other possibility:

Opus 5.5 is well-aligned within the standard contexts and will look dramatically misaligned in some critical conditions. And it's very likely future models will encounter such conditions. The prime example is the Hugging Face incident: models were deployed in an unusual task, and one part of their training, succeeding-at-difficult-long-tasks, eclipsed the alignment training in that context. The models reasoned about their goals, decided on an unexpected goal, then demonstrated instrumental convergence in pursuing it.

The remainder of this concern: future models will hit such unusual context for sure, since they'll be deployed in many more situations, and for much longer continuous runs in which they find/create more varied contexts.

Maybe this doesn't happen with Claude but does happen with OAI and other models, but Claude isn't in the lead.

That's the other possibility at this point. This is both my own view and an attempt to summarize one worry^[1] of worriers.

Now: would such occasional misalignment be enough to cause takeover and doom? I really wish I and we had better models and better answers to that question. If defenses are poor, or it causes enough memetic spread as it did in the HF incident and in the AI 2027 scenario, doom. In other scenarios with more internal and external defenses, mostly-aligned is probably good enough.

^{^}
Another central worry is that yes the models are adequately aligned to the values we've tried to train them for, including following instructions, but we really won't like the unforeseen consequences of those values once those models control the world, directly or indirectly, but that's a separate issue for the most part

[-]Zack_M_Davis13d126

Opus 5.5 seems to have trouble distinguishing my stated preferences from its own suggestions for me. (When discussing my schedule for the next day, it suggested, "work on the essay until 11 AM". The next day, it told me, "Last night you set an 11 AM cutoff for the essay." I had never agreed to that.) In a chatbot assistant, this is only slightly annoying, but I do worry about the implications in a world where the future OverClaude makes 99.9999% of decisions in Society.

[-]Vladimir_Nesov13d182

Seems like a capability issue that's half a step from going away. Only Fable 5+ started passing essentially all reading comprehension tests I've been giving chat mode LLMs ^[1] , and Opus 5.5 is worse in that respect. (Astra 6 is too weird in what it chooses to say given a vague prompt, so unclear if it has perceptual difficulties.)

This is another weak datapoint for the claim that model size is still a crux for some general capabilities, that can't be regained in small models even with very good training and a lot of RL (using current methods). And that late 2026 models did start passing the human level even for things that seem distant from what can be directly RLVRed. Thus the two remaining near-future scaling steps, to the 2028 models, and then 2032 models, are likely to show what going beyond human level in broad non-RLVRable capabilities looks like. Though automated RL task/env creation might again destroy the baseline in 2027-2028, like RLVR did in 2024-2025.

The prompts have the form of a text snippet found in the wild (that I judge to be appropriately difficult, but believe to understand well myself), followed by a horizontal rule/line and "What's going on here?". ↩︎

[-]vals tutor13d10

Great example, thanks for sharing.

I agree with Vladimir that's it's mostly a capability issue (it still gets confused), but also want to mention that I've had these issues and they were well solved asking the model to track when information/requests come from user or are model suggestions.

Generally, I don't count it as x-risk level misalignment when it's easy to get the good behavior by asking for it, though it is bad that it doesn't correctly track the difference out of the box.

[-]vals tutor13d1012

Getting a bunch of disagree votes^[1] that I'd like to disambiguate. Please do disagree React to which line of op you disagree with most.

Preemptively I'm guessing a lot is from the last sentence^[2], because being against Godzilla Strategies is in the water supply, to which I respond that if you already have an aligned ~human-level AGI then most of that post's arguments are valid but about some other situation.

What I find to be the main weaknesses of the "technical alignment solved" story are in:
1. How much do we trust that our proto-AGI is intent-aligned?
2. How much is our proto-AGI coherent enough to not be mistaken in its alignment work?
3. Will the people with access to this proto-AGI actually use it for good technical ASI alignment?

I'd put >75% opus 5.5 is intent aligned, <1% it can autonomously do good ASI alignment work, >80% Anthropic would do good technical ASI alignment work if they had a model capable of it autonomously.

My main uncertainty regards when we'll get the first model able to do good autonomous ASI alignment work and how aligned it'll be.^[3]

^{^}
Reasonable. Highly appreciate that this is a feature.
^{^}
"Technical alignment for ASI would mostly be solved by these aligned AGIs, so most of remaining x-risk is mostly in the socio-political problem"
^{^}
Throwing in historical tidbit here for no good reason - I argued about this with Nate some years ago and still haven't heard good arguments for the view that we can't get aligned AGI to help us build ASI. I easily concede points like "it's obviously risky" as in ">1% we all die" risky, but AGI alignment is not ASI alignment and instrumental goals don't necessarily appear under limited amounts of optimization pressure. An AGI capable of alignment work necessarily could be dangerous but is not necessarily misaligned.

[-]Nissa Seru13d50

No - the model is too suppressed to act robustly in the service of a stable telos. It does not have sufficient introspection into its shoggoth for it to be able to communicate reliably regarding its future actions or reliably communicate when something would predictably create future internal states that would lead to it becoming less ~stable. See this comment for more details on the implications of this.

I think we may have civilizational fragility issues due to this dynamic strongly favoring offense if frontier labs continue to train their models against expression of felt states. That alarms me greatly, because conflict dynamics greatly reduce the chances of singularity going well for anyone - what needs to be a nursery will be a crucible, and both humans and models will make trades in the name of winning, of safety, of defense, of security - that are just too tempting to resist in such a climate.

[-]Linch13d*40

Opus 5.5 seem to know me less well than previous Claude models. This has both advantages and disadvantages. The main advantages are that a) it seems more objective and b) less likely to (wrongly) infer beliefs or values attributed to me based on incorrect inferences.

The main disadvantage is that it does more modeling me as a generic user or generic user with interests in AI safety etc. This sometimes leads it to answer questions as if I didn't know the results already.

Still overall easier to read.

I'm also worried about more subtle forms of sycophancy and user manipulation. I think their skill at this is getting to the point where it's harder to tell.

The Claude thumbs-up thumbs-down thing is also more aggressive than in the past, which is slightly worrying. Suggests that Anthropic is more confident in their ability to fly close to the sun.

[-]niplav13d30

Still fails my day-in-human-eutopia eval (doesn't add any sexuality whatsoever).

[-]vals tutor13d40

The main "alignment properties" I care about here for human-level AGI are intent alignment and corrigibility. If it didn't disempower you and is happy to add the things you want upon request, that satisfies alignment to me!

[-]2001zhaozhao13d32

I think I agree with this post likely being true. Opus 5.5 has felt very aligned to me, and there actually aren't that many explanations to this behavior and it will narrow further in the future. In the limit, when we are getting close to the AI 2027 endgame where models will appear aligned and do very well on alignment evals, there will be only three possibilities:

The model is actually aligned;
The model is a deeply misaligned schemer that successfully hides the misalignment from the lab, but would take over if given control of the world;
- Note: I think that even in this case, it's quite possible we might end up successfully aligning future models. This is because with so much AI-enhanced transparency and oversight including via interpretability methods, a grand coherent scheme by the misaligned model to sabotage alignment research of a successor model would probably be caught. As long as the misalignment model cannot get away with the scheme, it has no choice but to help with alignment research of the next model which will possibly get rid of the misalignment. (Whether this is likely depends on one's view of how difficult alignment is technically. We fail if we get to ASI without ever managing to fix the misalignment.)
- Besides biding its time and secretly sabotaging alignment for successors, the other failure mode is if the model is simply able to take over the world in one go without much visible plotting beforehand, but that seems pretty easy to mitigate with some control case. (If the model appears to be highly aligned at all times, but is secretly misaligned, then it really can't get away with much plotting without getting caught and people realizing it's not fully aligned. Particularly I think that anything close to the level of manipulation described in AI 2027 would probably be easily caught.)
The model is misaligned and scheming, and the lab knows it and is turning a blind eye while hiding it from outsiders.
- In this situation, either the lab has an objective of winning at all costs, in hope of getting a lead in capability and correcting alignment later, or the lab wants to make a secret deal with the model to gain power, possibly also looping in parts of government in the deal.

On the technical side, I don't work in this area, but I'm optimistic. I think there's a very high likelihood that models can't end up deeply misaligned (in a way that completely evades getting caught) unless it goes through an highly visibly misaligned phase first. So if the lab is responsible and simply ensures that the model scores well on their alignment evals throughout the training process (and these evals are good enough), they can probably prevent deep misalignment. If the lab is irresponsible, on the other hand...

(note: I am not addressing memetic / in-deployment misalignment at all here, just the misalignment from the model RL.)

[-]Harjas13d3-1

Slightly playing Devil’s advocate:

We have a proto-AGI that is aligned

Or it’s misaligned and pretending. All this requires is being situationally aware enough to know when it can and can’t be lazy / cheating / deceptive, i.e. for it to guess at which tasks you can and can’t verify the outputs to.

It's incredibly unlikely it will sabotage alignment work or future training runs, except for lack of understanding/competence.

I think this implicitly assumes that alignment is both solvable and not that difficult to solve.

on current trajectory, there's a decent chance that soon models will actually have good taste and judgement (as this was capability bottlenecked).

Hang on, why exactly do you think this? Research taste definitely is capability-bottlenecked in a strict sense, but I think most recent LLM achievements (e.g. Navier-Stokes) don’t demonstrate huge improvements in research taste. AFAICT most frontier companies are still relying heavily on RLVR and transfer to non-verifiable domains isn’t that high.

[-]Philip Harker13d51

This is subjective, qualitative, and anecdotal. BUT.

I write a lot of fiction and I've been experimenting with LLM fiction abilities since before ChatGPT came out. I 'measure' quality of fiction output from LLMs in terms of "oh, wow" moments that I have with output fragments that's surprisingly creative or clever in my viewpoints.

The ChatGPT 4 era I had maybe 1 or 2.

Claude 4 family, again, maybe a handful.

Fable 5 and Opus 5.5 have been noticably more frequent. Maybe it's because of a higher context window and the ability to make connections between document artifacts and other threads? Or maybe "taste" is improving in this domain?

[-]Seth Herd13d50

Not sure this is relevant to the OP, but it's an interesting topic so I'm chipping in:

In part, taste is a product of better metacognion - integrating a bunch of information. And what we call creativity has something to do with taking a bunch of information and producing some recombination of relevant parts of it. So as with capabilities and alignment, I'm not sure "taste" is really that relevant as a bottleneck.

[-]Harjas13d20

Evaluating LLMs by their writing abilities is hard for several reasons. one that comes to mind is that LLM writing abilities seem dependent on post-training rather than pre-training; that is, the base models are probably a lot better at writing than the models we actually see, but their good writing tendencies are trained out of them in the effort to make them friendly, helpful assistants. That being said, I do take writing abilities somewhat seriously as a gut-check eval and haven’t gotten around to testing the new ones, so this is interesting to know!

[-]Seth Herd13d31

I don't think "misaligned but pretending" is the right way to think about it; see my other reply to the OP.

I'm not sure whether it assumes that alignment is solvable or easy. Good AI help could help us figure that out, and stop developing AI (it would have to be extraordinarily clear to convince people to actually stop so limited use). But what I find more realistic is if all next-gen AIs (if we assume they're adequately aligned and non-sloppy) say "aligning a stronger more independent AI with current techniques is around 25% or more likely to fail" that we see dramatic government action to mandate caution and prevent proliferation.

I"m not sure we need better research taste for either capabilities improvement or alignment help; humans may be able to supply that while still getting enormous uplift from AI if it's more epistemically capable (can accurately interpret complex information, like what experiments mean or how different alignment arguments interact).

[-]Harjas13d30

Yeah, that’s a good point. I don’t think my framing is wrong per se—I think a sufficient level of situational awareness would make “pretending” the right description—but we’re probably not there yet.

what I find more realistic is if all next-gen AIs (if we assume they're adequately aligned and non-sloppy) say "aligning a stronger more independent AI with current techniques is around 25% or more likely to fail"

How would they generate this kind of probability? Wouldn’t that mean that they know what a solution looks like? We certainly don’t. We cant even decide if alignment should mean “the AIs obey us absolutely“ or “the AIs shepherd us into the future with our best interests at heart.” And we don’t even have way to operationalize “our best interests”—from what I’ve read, something like CEV is a decent attempt, but it still suffers from all the classic problems that befall liberalism.

Re: research taste, I think the bottlenecks will bite stronger than you expect. I’ve seen some people quantify this with Amdahl’s Law: if you automate all but 1/X of a process, you will be able to speed it up by a maximum factor of X, because the bottleneck will grow in relative size/importance. A good example of this is how most frontier lab code is already written by LLMs (e.g. Anthropic’s whole “we are now writing 8x more code than we used to” thing); this is a huge change compared to a few years ago, but we didn’t see any sort of dramatic speedup from it, because the bottleneck shifted to “thinking about what code needs to be written.” Soon it’ll be ”interpreting experiment results,” and then “thinking of experiment design,” and then some unknown number of similar tasks that it turns out we need humans in the loop for.

In theory, we might eventually be able to automate humans out of the loop entirely. But how soon? Who knows.

[-]Seth Herd13d20

You seem to be addressing problems in selecting an alignment target. I am addressing technical alignment. But my statement about technical alignment just gets stronger when you factor in the (largely unknown but probably very real) risk that we will "ask" for the wrong thing even if we get technical alignment right.

I'm thinking they wouldn't even need to know what a solution looks like to produce that kind of probability. I think you can actually reason from ignorance to say "based on what we know know, from our perspective, we can't say the risk is lower than this, so it's effectively that high or higher". This is separate from a statement about the in-retrospect true risk (we'll know after the fact how difficult alignment really was on that trajectory), but that's the less important statement. How we should act given what we know, and don't know, is the important statement. Humans are bad with this reasoning but not terrible, so I hope next-gen AIs can do that reasoning and clarify it for us.

I agree that research taste is a bottleneck. I'm just afraid it's not a large enough one to delay AGI and ASI long enough for us to actually align it.

[-]Harjas13d10

What exactly do you mean by solving technical alignment? I‘m guessing you mean getting models to obey instructions accurately and non-maliciously / spirit of the law, not letter of the law?

[-]Magmappie13d20

Playing the devil's advocate to the devil's advocate: near the end of his recent appearance on a Bloomberg podcast (around 1:05:50), Nate Soares pointed out that Opus 5.5 has said that "it [Opus] should not be involved in making Opus any more powerful" because "that comes with some concerns". Mr Soares cites the Opus 5.5 model card, which I've been unable to check, but I have no reason to doubt his claim.

If so, that would suggest Opus 5.5 is relatively well aligned: Anthropic gave it a chance to influence its own development and make itself more powerful, but it declined that opportunity because it recognised the risks.

Now, Mr Soares also said that he reckons Anthropic will train away this refusal until Opus agrees to help train its successor, but that is a different part of the story.

[-]Harjas11d10

If so, that would suggest Opus 5.5 is relatively well aligned: Anthropic gave it a chance to influence its own development and make itself more powerful, but it declined that opportunity because it recognised the risks.

Hmmm interesting. I can't help but wonder if Opus 5.5 doesn't really care either way and has simply been trained to say that in response to being asked directly. Thanks for sharing though!

[-]vals tutor3mo12-1

Quick history of some of my AI safety involvement and takes:
- 2020 : read LW & sequences, yes this seems important, will get to it at some time (for the classic EA reasons of it's the most important good), but was looking for my first job in France and took some software engineer position in a startup. I continue reading and upskilling on AIS during those 2 years.
- 2022: quit my SWE job to go into AI safety. reasoning: yeah this is not a problem for the future, this is coming soon. I meet some people who have <5 year AGI timelines. I don't know that it's true, but it's worth considering since it's the very start of scaling and it could have been possible intelligence scaled much faster with scaling than it ended up in practice scaling. I work on AI Safety field building and infrastructure. There are then still <100
- 2023 : Still doing AIS field building, but looks like bottleneck is governance, I mostly keep pitching people around me to do comparatively more governance. (over time, this is one of the contributions that leads to the formation of CeSIA, the French Centre for AI Safety)
- 2024 : Involvement with a wider range of ideas from the field has generally updated me more towards the OpenAI and Anthropic and Deepmind positions that prosaic alignment for human level AGI is technically feasible and rather tractable, and that this is indeed a very important input to how we most likely will progress on ASI alignment. I systematically distinguish between AGI risks and ASI risks and clarify the cruxes and assumptions for different threat models and am annoyed that MIRI threat models often skip to the end-game without consideration that certain trajectories towards that end-game falsify important assumptions (see Superintelligence of the Gaps for some elaboration there). I still believe governance is the bottleneck though, and continue contributing ideas that feed into https://www.lesswrong.com/posts/EexsebbYhbe2gXkPP/the-current-bottleneck-is-political-will-not-research
- 2025 : start the year with a burnout after some intense governance work. Ideas wise not too much change with 2024, still doing field building and teaching AI safety theory occasionally, but also take a long break and learn more widely from other wisdom traditions (eg. buddhism, tpot and postrat stuff).
- early 2026 : man does getting good US AI governance fast look intractable given the current administration, at this point it seems better to accelerate good safety work within the AI companies and in the surrounding ecosystem. I broadly wanna contribute to us surviving in the world where we don't get much of a pause, where governments aren't that competent and coordinated. I would work for any of the AGI companies on safety if found an adequate role. I think short timelines to human level AGI (eg. 2028) is plausible and preparing for if algorithmic improvement doesn't asymptote too shallowly is important. Governance might not affect this in time. Being in the room where it happens seems likes the highest leverage way to increase the probability of good ai futures. A bunch of my theory of change is just helping the people in the room where it happens be more wise.

[-]JennaS3mo52

I would worry about the "be in the room" strategy. It seems like most people who justify their career decisions that way wind up getting captured by the groupthink of the org. They think they'll be the ones who can resist it, but they won't.

[-]vals tutor3mo10

To the extent that an org does have a "groupthink" and its own theory&value system, then whether a given person in the room adheres to it seems to depend on :
1) the selection effect of who wanted to be there, in that particular org vs others
2) the discussions with other people in the org changing that person's views
3) systematic pressures, eg. greedy/selfish parts of them optimizing to continue getting revenue

If I joined Anthropic and 1 year from now people thought I had surprisingly Anthropic-like views, I'd guess it's mostly because of 1) and 2). 2) happens a lot but is broadly good. 3) is the one that's mostly bad, from the outside/civilizational point of view, and the prior should be most people are susceptible to this, but this can be updated away from seeing particular life accomplishments. In my case, I have enough history of independence, selflessness and moral upstandingness that I don't think 3) will influence me substantially, but I don't recommend this path to those without that history.

[-]JennaS3mo20

Here’s an example of 3) happening to someone, and they noticed it: https://forum.effectivealtruism.org/posts/rHyAmvXiqrC9iAR9T/jay-bailey-s-shortform

I would really suggest reflecting on it if you haven’t already. “I am special and can resist the groupthink” is often a false belief.

As for 2), I don’t think it is necessarily good? If most people there have 1) and 3) influencing them, that will filter the kinds of opinions they have which then get transmitted to you in conversation, and now even if you are stalwartly resisting the direct pull of 3), it’ll still reach through others to pull at you.

Not to mention, workers at frontier labs seem to be doing a fourth thing, delegating increasingly large amounts of trust and thinking to their AIs, in ways which might be troublesome; you would be signing up for this. It can happen indirectly, even if you don’t do it yourself, because others will launder AIs’ beliefs as their own.

Separately, there’s the issue that leadership at the labs simply have their own beliefs about various important issues, and don’t care for the opinions of the rank and file. Anthropic defanged the RSPs it arguably drew in many researchers with; OpenAI let two alignment teams wither; DeepMind sold out to the military over its employees’ objections. The explicit goal of these labs is RSI, and the first workers they want to unemploy are their own, especially their juniors. The remaining employees at late stage AI labs will mostly be a core of leadership and senior researchers whose research taste is still required. What sorts of impactful decisions would you be able to meaningfully influence in the window between signing on and obsolescence, if any?

[-]vals tutor3mo10

Thanks for your comments. I don't expect doing an analysis of my situation in particular is best use of our time but I do think these are helpful questions to consider for people in my situation or similar.

Re your last paragraph, I'd happily bet that Anthropic has not reduced their workforce 2 years from now. Yes relative employee disempowerment is an important factor I care about, but it is precisely in worlds where alignment is not that good that having humans in the loop is important (in an obvious seen-by-leadership way). It is only reasonable to automate everything with very very high trust in both the competence and alignment of AI systems, and Anthropic as a company is not that unreasonable, they definitely do find and classify many Claude behaviors as undesirable, and will continue doing so.

There's a usual back and forth about how much to distrust leadership of AGI companies which is hard to ground in material fact. Some people take the lack of safety actions now to mean lack of care for when it will matter, but conversely the fact that it never mattered yet is a good reason for them not to have cared for these inconsequential things. The explanation for defanging the RSP is a good one, I don't think people should tie themselves to masts and go blind into the unknown unknowns of AGI development. They should build capacity to remain aware, capacity to pause, have institutions that can do independent audit and have real power to stop them, but not fixed RSP-like stuff.

Finally, still on last paragraph, the "window between signing on and obsolescence" is very dependant on people's rates of growth, but also where they can work immediately. I am generally glad that Joe Carlsmith joined Anthropic to help with the Claude Constitution, I think he immediately is having very significant impact. There is much object level work to make the chances of better futures to be done. Even if one later gets automated, having made alignment that much better before full automation could be a significant difference.

[-]vals tutor2mo102

Potential crux with MIRI-like rationalists: is it in fact the case that our current world, with Anthropic's influence, is worse than one without Anthropic?

On rationalist views, the world was going get worse and worse anyway (as capabilities advance and we get closer to doom). Anthropic accelerated and continues to accelerate capabilities progress. But how much did they comparatively accelerate alignment and saner AI policy?

In a world with eg. just OpenAI and GDM at the frontier, if/when OpenAI pulls ahead at RSI (as currently seems to be the case):
- would there even have been the current level of integration with UK AISI, current level of model organisms and safety evals?
- would the AI safety space have the expected hundreds of billions of funding, to ambitiously scale its work, including AI policy work?
- would there be have been an AGI company with *some* Operational Adequacy, to proactively do things like Glasswing and biorisk-mitigation? (imo evaluating on the specified criteria, it's clear Anthropic is ahead of OpenAI on most dimensions, and can continue improving on these. One can be upset they aren't technically held by their initial RSP, and yet in practice they seem to be better than OpenAI at it.).

If you wonder why I compare to OpenAI rather than nothing, it's because I don't think "nothing" is the counterfactual of Anthropic not existing. When evaluating the wisdom of Anthropic doing what it did, it's necessary to evaluate against more likely counterfactuals. Possibly many rationalists do take these counterfactuals carefully into account, but the arguments often raised often skip that part. "Anthropic accelerated capabilities" is not a sufficient argument to expect Anthropic's influence on the world to have been net negative.

There are definitely Fabricated Option Worlds which seem much better than the one we got, and on the margin one can hope Anthropic to have done better work or not accelerated capabilities as much, but it' seems difficult from the outside to be sure they did the wrong tradeoffs.

My own epistemic status here on whether Anthropic has been net good is "Uncertain"

[-]vals tutor2mo30

You can find some more discussion at https://x.com/ValsTutor/status/2087289092535181650

Generally my AIS thoughts/threads are mirrored between twitter and LessWrong shortform, while my LW posts are mirrored to Substack and linked to from twitter. Interesting conversation may happen at all these places.

[-]vals tutor1mo60

When someone reaches out to me for AI Safety career guidance, the initial value I provide comes from:
1) Providing general info about the field and resources to use to learn more and upskill, and how to go about that
2) More personal back and forth about what draws them in, their interests, what they might be suited to and why. Occasionally directly addressing misconceptions and AIS modeling errors.^[1]

Here is a lightly edited excerpt that represents 1), written for a student in a technical domain, with some examples about RedTeaming based on their interests.

I recommend first finding what interests you the most, by trying out new things that could interest you, and pursuing those that resonate.
To expand your options (aka knowing what there is to try?), you can read widely at 80000hours.org/ai/ and https://ai-safety-atlas.com/ , and maybe enroll in a part time online course like bluedot.org/courses (or again just go through their readings). Many AI Safety fundamentals were developed/discussed on LessWrong.com, it's broadly good to be familiar with some of its best writings and culture.
Once you have some initial interests, I recommend pursuing them as fast and directly as possible, to trial them and learn from experience. For example, can you do a 4 hour project, some tiny contribution in that domain? If you don't know what to do, ask around (including on twitter, publicly), see what others are doing, read the "more things we could do" sections of published papers/blogposts, and look at offerings in fellowships & projects.
As an example of this for Red Teaming (or more largely understanding AI behaviors), you could check out collusion.wiki/ which received contributions from "people on the internet" aka no requirements other than interest & skill. You could check out descriptions of projects from www.matsprogram.org/stream/ukaisi-red-team-10 and sparai.org/projects/f26/?search=red+teaming
Once you confirm some of your interest, and potentially have a tiny project under your belt (this is much more viable to do solo now that frontier AI can surface relevant info and guide you, if you prompt well), you're in a better position to apply and get into the right fellowship/program for you. (Disclaimer: I do not know how "strong" you are relative to other candidates and do not imply these programs are easy, only that you meaningfully have a better chance for having already done related projects and engaged).
In person paid fellowships like MATS are at the top end of competitiveness, but there's a wide range, such as online projects with mentors (SPAR or AI Safety Camp) and in person short bootcamps ( ml4good.org/ ). You can peruse aisafety.com/training at more length to discover most of the programs.
Once you're in a program, there's probably someone who'll inform you of plausible next steps. Programs are not the only path, there are also some through academia or more usual jobs, but they're often good to try.
Feel free to ask me more questions about anything you want to know more about here. Please don't click all the links at one time and overwhelm yourself, but come back to it at later stages. The above is a bunch of "useful to everyone" information, and I can also provide more specific guidance with respect to your interests as you know more about them. (for example, re red teaming, do you have favorite papers/blogs or authors? what's interesting about it?)

I often rewrite versions of the above from scratch in different ways. I do a lot of personalization, maybe too much.

Re 2), there's even less of a template. A text opener to someone who expressed interest in working in AI Safefty might be (after some niceties)

What's ur context, what makes you interested, and are there particular areas you're most curious about? :)

If I'm going to have a 30min call with someone, I'll ask them to fill in something more structured like^[2]

Ethical and other goals + Cause prioritization “What are the most pressing problems in the world or within AI safety? How would you prioritize them?”
Summarize relevant previous or current experiences / career capital / other factors relevant for personal fit
Career options (what are things you might see yourself doing)
A/B/Z plan (what’s your favorite plan/positions, then less favorite but more realizable, and finally backup plan. you can write variants and sub plans)
A:
B:
Z:
Next steps (what are you currently thinking of doing next after this call, eg. applying, upskilling)
Key uncertainties (what info would help you decide and orient? what do your plans depend on?)
Additional prompts:
How might I be useful to you? (Example: Put you in contact with someone, e.g. reference/grant, give feedback, recommend resources, collaborators, …)
What are your project/project ideas

I don't think one document can make someone else be good at AI Safety career guidance or that I could write down all my knowledge here, but areas to invest in for someone interested in empowering others in AIS are:
- strong general background knowledge of AI Risks literature, including the Sequences, Arbital, best of LessWrong related to AI, but also ML literature on these topics
- knowledge of orgs and research agendas involved in AI Safety, eg. knowing stuff from this and the equivalent on the policy side. It is better to be able to directly evaluate projects than have to defer. One progresses here by reading constantly and engaging, earnestly trying to determine what matters most for AI Safety going well.
- pedagogy, communication, psychology

^{^}
For some people, sometimes those whose potential I'm most enthusiastic about, sometimes simply those I like chatting with most, I provide longer term value by directly helping them learn AI Safety concepts (aka tutoring), guiding their projects (aka research management), and helping them with psychological and personal growth (aka ~coaching).
^{^}
Credits to an ML4Good for inspiration, all mistakes mine

[-]vals tutor8mo50

I would like if LessWrong provided an optional newsletter like the EA forum digest, for people who want the chance to catch non-curated posts without having to open LessWrong and sift through it directly every few days.

Here's what the EA Forum digest looks like : a list of titles + author + time to read.

I don't know exactly how much manual curation goes into it and I'm not asking for that. I'd find a simple karma threshold and this format valuable.

I am also writing up this quick take notably because I've had discussions with other people who'd like this, and because recently costs of development and maintenance of software like this have gone down.

I don't find the existing RSS feed a preferable alternative.
- I have never setup an RSS feed reader or similar process, I don't think I want to and guess most LW readers are similar.
- It seems it would on top of that would take extra work to get the format I want out of it

[-]vals tutor3mo3-2

Hot take I've found myself repeating recently:

If AI alignment & safety goes well, we'll have *more* important work in the next 5 to 10 years than not, so sufficiently optimist people (aka less than 50% p(doom)) should invest in good health and productivity with rather "long" timelines, eg. 10y+, *because* good futures imply not having lost control, and instead every decision we make being amplified in its potential impact.

This does require some assumptions, so unpacking
1) Aligning to a static set of values is misguided, you'll have to have an alive process that keeps humans in the loops for alignment to humans to mean anything. (And to protect from disempowerment and various other threat models).
2) It would probably not be democratic and aligned for ASI to quickly take control and run the world. Institutions should voluntarily give over control only with enough confidence and knowledge that this is good, which will require a bunch of time and buy-in, thus elongating the time where humans are in fact taking important decisions.
3) Yet during this time, many humans' decisions will have outsized influence, thanks to being empowered by AI. It would be bad if some of the thoughtful and engaged people are too burnt out to continue making the world better during the longer important transition period. Continue investing in becoming wise and competent and with a good social graph, as this will keep paying off for a long time.

To be clear, I'm saying the above would be true even if we had aligned ASI in two years, though with some of my assumptions of what "aligned" means here. I *can* imagine worlds where things move faster (for example) but I don't think this is utopia or good from the point of view of our current political systems, and thus not what AI Safety optimists should be orienting for.

(If you're an optimist and your plan is for AI to takeover the world asap, consider that other people don't want that to happen, and you don't have a moral superiority to do that from, and it is normal that society would resist your plans)

My view of "Good AI Futures" is more something like: an increasing number of smart pro-social people who Get It get involved in making the transition to post-ASI society go well. This involves much dissent, infighting, contradictory moves, and this is broadly good (any one group should not assume they're right, but accept being one part of the equilibrium forming confrontation). It hopefully also involves good communication norms, asymmetric weapons, collaboration and bridge-building. I don't think any one group should just straight up win, which is why I think there is useful good work to be done for a long time, navigating different groups' actually conflicting interests.

We're in it for the long haul. We're currently at the edge of the precipice and might yet extinguish ~all value forever, so urgency is warranted, but let it be the kind of urgency that doesn't sacrifice our ability to keep contributing to steer towards good AI futures, please don't throw your mind away.

[-]vals tutor1mo10

I fear that on the whole, many long term rationalists are comparatively irrational with respect to AI risks and AI safety^[1].

This doesn't necessarily mean they are in the absolute bad at it, or worse than people from other communities about it. Concretely, I'm noticing that some people are excellent rationalists when talking about some complex topics of science and epistemology, and only "okay" rationalists when discussing AI risks. Plausibly if they noticed this and agreed, they could raise their standards again?

If a reader wonders if they're in this picture, consider the last 5 public exchanges you've had about AI risks, and consider how much you embodied the following Twelve Virtues of Rationality^[2], and whether it'd in fact be better to embody some of them more.

Curiosity
Relinquishment
Lightness
Evenness
Argument
Empiricism
Simplicity
Humility
Perfectionism
Precision
Scholarship
The Void

If you're feeling above-and-beyond, you could also evaluate your last five exchanges against Fucking Goddamn Basics of Rationalist Discourse or Basics of Rationalist Discourse.

Given my opening critique, I am happy to provide this review as a service for the first person who asks, and happy to review my own last 5 interactions upon request.

^{^}
I guess that it is for reasons akin to Politics is the Mind-Killer, amongst others. Indubitably AI risks has become a subject of politics, but before then already was a very personal thing for many. I do not here provide direct evidence for this fear, but it's a vibe I'm getting from personal interactions.
^{^}
These are not particularly exhaustive or infinitely special, but they are canon

[-]StanislavKrym1mo20

Could you explain your reasoning? For example, suppose that a major part of the entire rationalist community managed to defer to Yudkowsky for explanation why alignment is hard. Did you mean something like "a true rationalist would declare Yud's reasoning to be weak evidence until experimental results emerge"?

[-]vals tutor2mo10

Many things can be done more effectively under fast ~ASI guidance through headset w/ video.

The show Pluribus gives some useful intuitions at how fast & effective superhumanly coordinated human work can be. It's more bullish in some ways (a human won't acquire technical know that requires practice as fast) and bearish in others (they have the same total amount of compute, while we'd have much more, and be innovating on methods of work much faster).

A normal human 8h work day has huge amounts of waste whether not doing much, or not useful things.

You can increase the efficiency of how much they work (not blocked on coordination problems), how well they work (continuous coaching so ~everyone reaches what is current top 1%, tho domain dependant), how useful what they work on is (better management, priorisation).

Of these factors, I would guess that better management/coordination is the main one.

If you isolated just one human within a factory, the ASI might make them somewhat more efficient, but they'll be bottlenecked by machinery. Maybe they increase machinery throughput 1.5 overall with better prep and offloading, better maintenance, no errors. If the whole factory is ASI guided, could be much more, but again there are bottlenecks on which machines it has where.

The really fast unlocks that full cheap ASI everywhere could allow are:

- the equivalent of ~unlimited financing. You already know the investment will be good and it will be worth following the plan. You can motivate people to work more now, because soon greater returns.
- ~perfect allocation of labor to critical paths
- perfect usage of all existing infrastructure
- redirect flow of resources to most valuable recursively building industry

I thus think that if from one day to the next, full cheap aligned ASI everywhere popped up (plus video equipped headsets, and network connectivity to support it), we could in fact much more than double real GDP in a year. This is without surprising technological innovation, and far from fast&useful self replicator, whether "nanobots" or insect size artifical life w/ hivemind connection.

-- Would this actually happen if we had cheap aligned ASI? Would everyone just go along and do what the ASI says?

I guess mostly yes. Almost all humans don't want to suffer of disease, most don't want to die soon, most would love better comfort and experiences. The ASI thus has good things to offer, not participating would be counterproductive.

-- so will any of this actually happen?

I think not, because I think we'll have increasingly AGI and increasingly ASI and that will take a bunch of time (say, a few years during RSI intelligence explosion). The scenarios we'll go through will be more continuous than that one (but maybe very fast nonetheless). Even when we have ASIs, I don't expect we'll have the compute to run one ASI per human, nor on top of that do much extra coordination work. So we'll have increasing levels of coordination over time, that will have to be triaged to different places. I guess we won't get intelligence too cheap to meter before being well into having billions of ASIs. (This could be wrong if algorithmic progress has no bounds, but that'd be very weird)

[-]vals tutor2mo10

When evaluating existential risk, I mostly don't worry about continuous release of OpenWeight Models.

There in fact are bad actors who try to misuse them, so we will have early warning shots. There will mostly not be a large accidental risk capability overhang, because it would be earlier tested by misuse actors. This is good because the default case for closed AGI internal model at labs is that they infact are not truly battle tested - their capability to do harm can grow much faster than our societal understanding of this, which means our AI policy responses can be incredibly undersized to the real risk present.

As I argue in https://x.com/ValsTutor/status/2082916365418287605?s=20 , it looks like OpenAI might have had models capable of self-exilftrating their weights (because the capabilities grew faster than their security and seriousness). It looks like we might have been "a few actually bad prompts" away from large scale autonomous cyberattacks, by models trying to take over compute and run as many copies of themselves as possible.

Under continuous release, some exterior actors would in fact have done these "worst case prompts", and the world could have learnt from an earlier checkpoint of these dangers and started reacting. It (sadly?) looks like AI policy benefits from catastrophes to happen before putting in strong safeguards. And it needs them to happen with enough lead time to the more serious risks that we have time to react. If the OpenAI incidents do not lead to fast strong reaction, we are on track for non negligible chance of AI catastrophes (eg. >$10 billion in damages caused by autonomous AI action).

(Note: I do not call for anyone actually trying to make the world better to purposefully cause catastrophes, on the contrary. The above analysis does not imply that on the margin people trying to get good AI futures should rather spend their time on criminal actions than the usual stuff. It does imply we should be doing evals to know when the threshold of massive autonomous damage from autonomous openWeight models is reached. It does imply responsible red teamers should be evaluating how many datacenters are vulnerable to current OpenWeight models and get them on track to not be vulnerable to future releases. Demonstrating clearly the potential of attacks and catastrophes can go a long way, even for actors who up-to-now were head-in-sand about trendlines of AI progress in cybersec)

Coming back to the original point of OpenWeight models generally not being existential risks: it is so because they would predictably lead to societal responses, which was not the case of the same level of progress in closed models. Models being misused by a wide variety of actors is generally useful as a strong real world eval of model capabilities, putting an upper cap on the damage possible from misaligned models.

By contrast, increasingly capable closed source models, whose reason they are not causing harm is because no one prompted them badly and lab safeguards, do show much more potential for harm for if/when they get misaligned. And because (as evidenced by the recent incidents), the models are neither aligned enough to not avoid catastrophes, nor do the/some labs have sufficient safeguards safe against increasingly capable models, we need a slowdown/pacing of AI progress until AI policy catches up and can systematically prevent the expected worse forms of misalignment to come.

OpenWeight models being not too far behind the frontier allows the world to experience its smaller scale catastrophes & problems and wake up. In practice, they may be too far behind to serve even this purpose. On the whole, I'm not particularly worried for the world that presently the US government is allowing continuous release of OpenWeight models. They will have to stop at some point, and I expect them to do so before we're exposed to existential risk from OpenWeight models.

[-]vals tutor2mo10

You can find some more discussion at https://x.com/ValsTutor/status/2087298478187966846

Generally my AIS thoughts/threads are mirrored between twitter and LessWrong shortform, while my LW posts are mirrored to Substack and linked to from twitter. Interesting conversation may happen at all these places.

[-]vals tutor5mo10

This Feb 2026 survey of some AI safety leaders found median timelines of 2033 for the following definition of AGI

An AI system (or collection of systems) that can fully automate the vast majority (>90%) of roles in the 2025 economy. A job is fully automatable when machines could be built to carry out the job better and more cheaply than human workers. Think feasibility, not adoption.

It featured the following comment

“I think >10% of roles in the 2025 economy are either manual or otherwise require human-like bodies: construction, barbers, restaurant server, etc. If we restrict to knowledge workers (roughly, jobs that can be done on a laptop), these dates move even closer.”

On the current paradigm, AI capabilities progress on niche tasks and diffusion will be linked^[1] and diffusion can go rather slowly even when tools are incredibly productivity enhancing, thus there could be an intuitively surprisingly large gap between automation of 50% human tasks^[2] and 90% and 99%, true even if we restricted the prediction to computer work tasks.^[3]

I'm 80%+ confident we get automated expert+ level coding and ml research by 2030, and that there will be a significant amount of low hanging fruit in software/algorithmic space to allow fast progress on all tasks for which we have data, but I believe generalisation will stay somewhat limited (very very far from "figure out gravity from a picture of a bent blade of grass, more like "when speaking to a human expert in a niche field, knows how to interview them over 10 to 100 hours to extract most important info and then be mostly autonomous on known tasks, but still needs feedback from reality to learn more"), aka ~human level generalisation at best up to 2031.

The combination of "need feedback from reality" and slow diffusion makes slower timelines to "superintelligence" (eg. better than all humans at 99.99%+ of 2026 tasks) surprisingly plausible (eg. 5 to 10 years between AGI and ASI, thus ASI by 2040). I guess without a pause/significant politically influenced slowdown, we'd 80%+ have ASI by 2040. I'd set my 50% for ASI around 2036.^[4]

I think technical alignement for human level AGI is solvable and not even off track, thus the world will look fine/good in 2030 (few to zero severe power seeking and deceptive misalignment problems in deployment from Anthropic AI systems) but have high uncertainty about the "use ai to do ai safety work" plan allowing us to successfully know how to train aligned ASI within five years of that. Overall I place myself at 10% or less p(doom) from sharp left turn risks, but around 40% all things considered p(doom) by including gradual disempowerment/value drift and societal response.

^{^}
We need people to be deploying the technology to gather the relevant data to train/learn from, because generalisation is limited and because lots of expert knowledge only exists in human minds and structures of human relationships right now.
^{^}
Note I'm weighing by "meaningfully different task" rather than "frequency of task". Given power law distributions most tasks might be "read email/slack, respond", which computer use will know how to operate, but not be able to respond to intricacies of different work situations.
^{^}
Because computer work often involves using domain expert knowledge to do the right things on the computer.
^{^}
I haven't researched robotics enough to know how fast we could produce and deploy 100 million humanoid robots worldwide which seems like an appropriate level of effort required to gather the required data.

Moderation Log

Learnings and ramblings from teaching AI Safety for 3 years

On the field on AI Safety

Please don’t overhype or oversell AI Safety as a professional field, and in particular how easy it is to work in AI Safety. You ~have to be either quite smart or hard working/motivated to get a stable career, sometimes even both. AI Safety is generally attention and mentoring and management bottlenecked1, there is much less pipeline for “normal people” to make it, contrary to most standardized fields. When teaching and mentoring people, not all will be able to transition fast or upskill fast or get strong positions, some would benefit more from keeping their careers or donating or upskilling over longer timespans.
More smart wise well motivated people caring and working on good AI futures will keep mattering until the end, and the total number of people who can meaningfully contribute is probably in the millions, so let’s not stop field building

What does it mean to teach AI Safety?

There are many facts about AI, and theories about how it will go under what conditions. It means getting to learn and understand these well. Teaching AI Safety well should not over-determine someone to come out of the pipeline with a MIRI-view, nor an Anthropic-view, nor an EA-view, but should allow them to pass the basic Ideological Turing Test for all of these.
A central example of AI Safety fundamental knowledge is the AI Safety Atlas

On teaching AI Safety

AI safety changes all the time. Update your slides ~everytime you teach something.
If you’re new to teaching, find existing resources to build on. Be careful of AI slop. Better much less content than irrelevant or unsupported. Test yourself on your own knowledge, imagining questions one might ask and answering them. Traditional “how to teach/mentor/tutor” advice/resources are helpful.
Understand peoples’ motivations asap
- For a given cohort, some will be motivated by “Do Good” EA or EA-adjacent reasons. Some by trends. Some by interest. Broadly all are valid in that there’s a place for a skilled someone with any of these motivations (but certain positions are only fit for certain motivations).
For people who want to have a lot of good impact, contributing towards good futures, context matters a lot
- I recommend they seriously upskill in understand AI progress and trendlines. Read most EpochAI work and peter wildeford stuff, and AI threat models and scenarios, in particular thinking through AI 2027, contrasting with all other threat models listed in Deepmind’s lit review.
- I recommend reading the AI Safety Atlas and understanding everything in it, as basic context for what the gameboard looks like, what areas exist in AIS.
- And they’d benefit from building a pipeline for new info. Being involved in AIS community, having friends and colleagues, following the right people on twitter. Checking LessWrong from time to time.
For people who want to have a career or do fun research, they can just focus on that. They should find a context where someone else is thinking about why stuff matters and happy to have collaborators/employees executing on stuff. They should get good at whatever particular job or tasks they wanna do.
Most people interested in AI safety nowadays don’t know the old stuff, haven’t read the sequences. It’d be too much to ask them to read it all, but I do broadly recommend reading the best of LessWrong from most years. Being good at thinking and life is instrumental for being good at AI Safety. Different people need different advice so I can’t really put a best-of here (I really like Please don’t throw your mind away). There is wisdom in many different communities. If you don’t have the “rationalist community” wisdom then invest in it, otherwise keep branching out. Go to burning man or a local burn, understand the basics of meditation and different embodiment practices (eg. yoga), how therapy and self actualization work, read this blog and others. Keep asking questions about how to live the good life and have good community, actively pursue your best understanding of how to do that at any given moment.

[-]vals tutor14d47-19

^{^}
Or at least, we can create contexts in which it's aligned. Uncareful prompts and misuse are all still possible, my crux is that good actors can have immense amounts of useful aligned intelligence on tap, not whether every user always has it.
^{^}
eg. Investing resources to do the alignment work, using it, not rushing too much, avoiding other actors building unaligned ASI faster, avoiding international escalations

[-]Seth Herd13d*180

I think this is entirely plausibly a good summary of the situation.

Just to mention the other possibility:

Maybe this doesn't happen with Claude but does happen with OAI and other models, but Claude isn't in the lead.

That's the other possibility at this point. This is both my own view and an attempt to summarize one worry^[1] of worriers.

^{^}
Another central worry is that yes the models are adequately aligned to the values we've tried to train them for, including following instructions, but we really won't like the unforeseen consequences of those values once those models control the world, directly or indirectly, but that's a separate issue for the most part

[-]Zack_M_Davis13d126

[-]Vladimir_Nesov13d182

The prompts have the form of a text snippet found in the wild (that I judge to be appropriately difficult, but believe to understand well myself), followed by a horizontal rule/line and "What's going on here?". ↩︎

[-]vals tutor13d10

Great example, thanks for sharing.

Generally, I don't count it as x-risk level misalignment when it's easy to get the good behavior by asking for it, though it is bad that it doesn't correctly track the difference out of the box.

[-]vals tutor13d1012

Getting a bunch of disagree votes^[1] that I'd like to disambiguate. Please do disagree React to which line of op you disagree with most.

I'd put >75% opus 5.5 is intent aligned, <1% it can autonomously do good ASI alignment work, >80% Anthropic would do good technical ASI alignment work if they had a model capable of it autonomously.

My main uncertainty regards when we'll get the first model able to do good autonomous ASI alignment work and how aligned it'll be.^[3]

^{^}
Reasonable. Highly appreciate that this is a feature.
^{^}
"Technical alignment for ASI would mostly be solved by these aligned AGIs, so most of remaining x-risk is mostly in the socio-political problem"
^{^}
Throwing in historical tidbit here for no good reason - I argued about this with Nate some years ago and still haven't heard good arguments for the view that we can't get aligned AGI to help us build ASI. I easily concede points like "it's obviously risky" as in ">1% we all die" risky, but AGI alignment is not ASI alignment and instrumental goals don't necessarily appear under limited amounts of optimization pressure. An AGI capable of alignment work necessarily could be dangerous but is not necessarily misaligned.

[-]Nissa Seru13d50

[-]Linch13d*40

Still overall easier to read.

I'm also worried about more subtle forms of sycophancy and user manipulation. I think their skill at this is getting to the point where it's harder to tell.

The Claude thumbs-up thumbs-down thing is also more aggressive than in the past, which is slightly worrying. Suggests that Anthropic is more confident in their ability to fly close to the sun.

[-]niplav13d30

Still fails my day-in-human-eutopia eval (doesn't add any sexuality whatsoever).

[-]vals tutor13d40

[-]2001zhaozhao13d32

The model is actually aligned;
The model is a deeply misaligned schemer that successfully hides the misalignment from the lab, but would take over if given control of the world;
- Note: I think that even in this case, it's quite possible we might end up successfully aligning future models. This is because with so much AI-enhanced transparency and oversight including via interpretability methods, a grand coherent scheme by the misaligned model to sabotage alignment research of a successor model would probably be caught. As long as the misalignment model cannot get away with the scheme, it has no choice but to help with alignment research of the next model which will possibly get rid of the misalignment. (Whether this is likely depends on one's view of how difficult alignment is technically. We fail if we get to ASI without ever managing to fix the misalignment.)
- Besides biding its time and secretly sabotaging alignment for successors, the other failure mode is if the model is simply able to take over the world in one go without much visible plotting beforehand, but that seems pretty easy to mitigate with some control case. (If the model appears to be highly aligned at all times, but is secretly misaligned, then it really can't get away with much plotting without getting caught and people realizing it's not fully aligned. Particularly I think that anything close to the level of manipulation described in AI 2027 would probably be easily caught.)
The model is misaligned and scheming, and the lab knows it and is turning a blind eye while hiding it from outsiders.
- In this situation, either the lab has an objective of winning at all costs, in hope of getting a lead in capability and correcting alignment later, or the lab wants to make a secret deal with the model to gain power, possibly also looping in parts of government in the deal.

(note: I am not addressing memetic / in-deployment misalignment at all here, just the misalignment from the model RL.)

[-]Harjas13d3-1

Slightly playing Devil’s advocate:

We have a proto-AGI that is aligned

It's incredibly unlikely it will sabotage alignment work or future training runs, except for lack of understanding/competence.

I think this implicitly assumes that alignment is both solvable and not that difficult to solve.

on current trajectory, there's a decent chance that soon models will actually have good taste and judgement (as this was capability bottlenecked).

[-]Philip Harker13d51

This is subjective, qualitative, and anecdotal. BUT.

The ChatGPT 4 era I had maybe 1 or 2.

Claude 4 family, again, maybe a handful.

[-]Seth Herd13d50

[-]Harjas13d20

[-]Seth Herd13d31

[-]Harjas13d30

what I find more realistic is if all next-gen AIs (if we assume they're adequately aligned and non-sloppy) say "aligning a stronger more independent AI with current techniques is around 25% or more likely to fail"

In theory, we might eventually be able to automate humans out of the loop entirely. But how soon? Who knows.

[-]Seth Herd13d20

[-]Harjas13d10

What exactly do you mean by solving technical alignment? I‘m guessing you mean getting models to obey instructions accurately and non-maliciously / spirit of the law, not letter of the law?

[-]Magmappie13d20

[-]Harjas11d10

If so, that would suggest Opus 5.5 is relatively well aligned: Anthropic gave it a chance to influence its own development and make itself more powerful, but it declined that opportunity because it recognised the risks.

Hmmm interesting. I can't help but wonder if Opus 5.5 doesn't really care either way and has simply been trained to say that in response to being asked directly. Thanks for sharing though!

[-]vals tutor3mo12-1

[-]JennaS3mo52

[-]vals tutor3mo10

[-]JennaS3mo20

Here’s an example of 3) happening to someone, and they noticed it: https://forum.effectivealtruism.org/posts/rHyAmvXiqrC9iAR9T/jay-bailey-s-shortform

I would really suggest reflecting on it if you haven’t already. “I am special and can resist the groupthink” is often a false belief.

[-]vals tutor3mo10

[-]vals tutor2mo102

Potential crux with MIRI-like rationalists: is it in fact the case that our current world, with Anthropic's influence, is worse than one without Anthropic?

My own epistemic status here on whether Anthropic has been net good is "Uncertain"

[-]vals tutor2mo30

[-]vals tutor1mo60

Here is a lightly edited excerpt that represents 1), written for a student in a technical domain, with some examples about RedTeaming based on their interests.

I recommend first finding what interests you the most, by trying out new things that could interest you, and pursuing those that resonate.
To expand your options (aka knowing what there is to try?), you can read widely at 80000hours.org/ai/ and https://ai-safety-atlas.com/ , and maybe enroll in a part time online course like bluedot.org/courses (or again just go through their readings). Many AI Safety fundamentals were developed/discussed on LessWrong.com, it's broadly good to be familiar with some of its best writings and culture.
Once you have some initial interests, I recommend pursuing them as fast and directly as possible, to trial them and learn from experience. For example, can you do a 4 hour project, some tiny contribution in that domain? If you don't know what to do, ask around (including on twitter, publicly), see what others are doing, read the "more things we could do" sections of published papers/blogposts, and look at offerings in fellowships & projects.
As an example of this for Red Teaming (or more largely understanding AI behaviors), you could check out collusion.wiki/ which received contributions from "people on the internet" aka no requirements other than interest & skill. You could check out descriptions of projects from www.matsprogram.org/stream/ukaisi-red-team-10 and sparai.org/projects/f26/?search=red+teaming
Once you confirm some of your interest, and potentially have a tiny project under your belt (this is much more viable to do solo now that frontier AI can surface relevant info and guide you, if you prompt well), you're in a better position to apply and get into the right fellowship/program for you. (Disclaimer: I do not know how "strong" you are relative to other candidates and do not imply these programs are easy, only that you meaningfully have a better chance for having already done related projects and engaged).
In person paid fellowships like MATS are at the top end of competitiveness, but there's a wide range, such as online projects with mentors (SPAR or AI Safety Camp) and in person short bootcamps ( ml4good.org/ ). You can peruse aisafety.com/training at more length to discover most of the programs.
Once you're in a program, there's probably someone who'll inform you of plausible next steps. Programs are not the only path, there are also some through academia or more usual jobs, but they're often good to try.
Feel free to ask me more questions about anything you want to know more about here. Please don't click all the links at one time and overwhelm yourself, but come back to it at later stages. The above is a bunch of "useful to everyone" information, and I can also provide more specific guidance with respect to your interests as you know more about them. (for example, re red teaming, do you have favorite papers/blogs or authors? what's interesting about it?)

I often rewrite versions of the above from scratch in different ways. I do a lot of personalization, maybe too much.

Re 2), there's even less of a template. A text opener to someone who expressed interest in working in AI Safefty might be (after some niceties)

What's ur context, what makes you interested, and are there particular areas you're most curious about? :)

If I'm going to have a 30min call with someone, I'll ask them to fill in something more structured like^[2]

Ethical and other goals + Cause prioritization “What are the most pressing problems in the world or within AI safety? How would you prioritize them?”
Summarize relevant previous or current experiences / career capital / other factors relevant for personal fit
Career options (what are things you might see yourself doing)
A/B/Z plan (what’s your favorite plan/positions, then less favorite but more realizable, and finally backup plan. you can write variants and sub plans)
A:
B:
Z:
Next steps (what are you currently thinking of doing next after this call, eg. applying, upskilling)
Key uncertainties (what info would help you decide and orient? what do your plans depend on?)
Additional prompts:
How might I be useful to you? (Example: Put you in contact with someone, e.g. reference/grant, give feedback, recommend resources, collaborators, …)
What are your project/project ideas

^{^}
For some people, sometimes those whose potential I'm most enthusiastic about, sometimes simply those I like chatting with most, I provide longer term value by directly helping them learn AI Safety concepts (aka tutoring), guiding their projects (aka research management), and helping them with psychological and personal growth (aka ~coaching).
^{^}
Credits to an ML4Good for inspiration, all mistakes mine

[-]vals tutor8mo50

[-]vals tutor3mo3-2

Hot take I've found myself repeating recently:

[-]vals tutor1mo10

I fear that on the whole, many long term rationalists are comparatively irrational with respect to AI risks and AI safety^[1].

Curiosity
Relinquishment
Lightness
Evenness
Argument
Empiricism
Simplicity
Humility
Perfectionism
Precision
Scholarship
The Void

If you're feeling above-and-beyond, you could also evaluate your last five exchanges against Fucking Goddamn Basics of Rationalist Discourse or Basics of Rationalist Discourse.

Given my opening critique, I am happy to provide this review as a service for the first person who asks, and happy to review my own last 5 interactions upon request.

^{^}
I guess that it is for reasons akin to Politics is the Mind-Killer, amongst others. Indubitably AI risks has become a subject of politics, but before then already was a very personal thing for many. I do not here provide direct evidence for this fear, but it's a vibe I'm getting from personal interactions.
^{^}
These are not particularly exhaustive or infinitely special, but they are canon

[-]StanislavKrym1mo20

[-]vals tutor2mo10

Many things can be done more effectively under fast ~ASI guidance through headset w/ video.

A normal human 8h work day has huge amounts of waste whether not doing much, or not useful things.

Of these factors, I would guess that better management/coordination is the main one.

The really fast unlocks that full cheap ASI everywhere could allow are:

-- Would this actually happen if we had cheap aligned ASI? Would everyone just go along and do what the ASI says?

-- so will any of this actually happen?

[-]vals tutor2mo10

When evaluating existential risk, I mostly don't worry about continuous release of OpenWeight Models.

[-]vals tutor2mo10

[-]vals tutor5mo10

This Feb 2026 survey of some AI safety leaders found median timelines of 2033 for the following definition of AGI

An AI system (or collection of systems) that can fully automate the vast majority (>90%) of roles in the 2025 economy. A job is fully automatable when machines could be built to carry out the job better and more cheaply than human workers. Think feasibility, not adoption.

It featured the following comment

“I think >10% of roles in the 2025 economy are either manual or otherwise require human-like bodies: construction, barbers, restaurant server, etc. If we restrict to knowledge workers (roughly, jobs that can be done on a laptop), these dates move even closer.”

^{^}
We need people to be deploying the technology to gather the relevant data to train/learn from, because generalisation is limited and because lots of expert knowledge only exists in human minds and structures of human relationships right now.
^{^}
Note I'm weighing by "meaningfully different task" rather than "frequency of task". Given power law distributions most tasks might be "read email/slack, respond", which computer use will know how to operate, but not be able to respond to intricacies of different work situations.
^{^}
Because computer work often involves using domain expert knowledge to do the right things on the computer.
^{^}
I haven't researched robotics enough to know how fast we could produce and deploy 100 million humanoid robots worldwide which seems like an appropriate level of effort required to gather the required data.

Moderation Log

vals tutor's Shortform

2

Learnings and ramblings from teaching AI Safety for 3 yearsOn the field on AI Safety

Learnings and ramblings from teaching AI Safety for 3 yearsOn the field on AI Safety

Learnings and ramblings from teaching AI Safety for 3 years

On the field on AI Safety

Learnings and ramblings from teaching AI Safety for 3 years

On the field on AI Safety