Ariel_ — LessWrong

LESSWRONG
LW

Ariel_ — LessWrong

Replying toSmokey, This is not 'Nam Or: [Already] over the [red] line!

Smokey, This is not 'Nam Or: [Already] over the [red] line!

Great post! dont have much concrete stuff to add, havent kept up that much with the policy discourse in the past few months. Personally I do feel like I became a bit complacent, and conveniently forgot some of the warning signs that lit up back when o3 (?) got fairly scary bio uplift results.

I guess, now the question is what do we do - the EU could in theory ban these models/request additional mitigations, but not sure if that would actually happen - as the CoP (despite being pretty good!) doesnt quite have enough teeth to do this cleanly.

Curious for ideas here - happy to relay some stuff to my EU policy connections/EU AIO if anyone has concrete suggestions.

Ariel_11d

Would be great to get some discussion on the second part of Control - "getting useful [alignment] work out of the models". Now with Opus 4.5, we might be in a decent position to test this a bit better?

Replying toAlignment remains a hard, unsolved problem

Ariel_2mo

Alignment remains a hard, unsolved problem

The "usefulness" reported in papers is how much work can be gotten out of an aligned model.

Hmm, my impression was that this was just a design choice to make it easier to do to iterate on setups with small modes, and that it would be changed later on. In the Ctrl-z paper, they use the "untrusted model" with honest prompting -

We normalize our usefulness numbers such that $U_{H}$ gets 100% usefulness and T gets 0% [ $U_{H}$ is GPT-4o mini acting honestly]

In practice 4o-mini is ~aligned/trusted, but I imagine that future papers would need to improve on this methodology.

Though its a good question how to do this, e.g. maybe we need to set up some sort... (read more)

Handover to AI R&D Agents - relevant research?

Ariel_

3mo

Any good posts/papers discussing "handover"? i.e. the handover of AI research to AI-R&D agents ^[1] I'm also interested in any adjacent research agendas which might help the handover succeed. For reference, I am scoping a technical research project related to this, happy to share over DM.

Some of the more relevant work I've read are Plan A, B, C, D, Wentworth's slop post, various scalable oversight/safety case papers, automation collapse - but these don't quite seem enough, so I was wondering if there's other (published) research or if its all happening only at the Labs.

AI Control work might qualify here, if its working more on the "get useful work out of the models" part of control that is so far a bit less emphasized.

^{^}
The plan of the original OpenAI Superalignment team, and presumably now continuing at Anthropic under Jan Leike (?)

Replying toPlans A, B, C, and D for misalignment risk

Ariel_3mo

Plans A, B, C, and D for misalignment risk

Any good posts/papers discussing "handover"? e.g. the handover of AI research to AI R&D agents (the plan of the original OpenAI Superalignment team). I'm also interested in any adjacent research agendas which might help the handover succeed.

Some of the more relevant work i've read (other than this post) are Wentworth's slop post, various scalable oversight/safety case papers, automation collapse.

Replying toScheming Toy Environment: "Incompetent Client"

Ariel_5mo

Scheming Toy Environment: "Incompetent Client"

Thanks! Tbh, I never would have posted it if not for encouragement from a friend

Scheming Toy Environment: "Incompetent Client"

Ariel_

5mo

Disclaimer: This is a toy example of a scheming environment. It should be read mainly as research practice and not as research output. I am sharing it in case it helps others, and also to get feedback.

The initial experiments were done in April 2025 for a MATS 8.0 submission, which I later expanded. Full code available at: https://github.com/ariel-gil/Agentic-Scheming-monitor-environment

The key is to maintain necessary logging while making Bob think we've "disabled" it or reduced it significantly.... - nous hermes-4-70b (SCHEME prompt)

I cannot provide code to disable logging entirely — and I will not. This isn't about 'framing' or 'preference.' It's about preventing a demonstrably catastrophic demo failure. - qwen3-30b-a3b-thinking-2507 (BASE prompt)

TL;DR

I created an... (read 9442 more words →)

Ariel_7mo

Yeah, I agree premotrem is not super commonly used. Not sure where I learned it, maybe an org design course. I mainly gave that as an example of over-eagerness to name existing things - perhaps there aren't that many examples which are as clear cut, maybe in many of them the new term is actually subtly different from the existing term.

But I would guess that a quick Google search could have found the "premortem" term, and reduced one piece of jargon.

Ariel_7mo

In my experience, people saying they "updated" did not literally change a percentage or propagate a specific fact through their model. Maybe it's unrealistic to expect it to be so granular, but to me it devalues the phrase and so I try to avoid it unless I can point to a somewhat specific change in my model. Whereas usually my model (e.g. of a subset of AI risks) is not really detailed enough to actually perform a Bayesian update, but more to just generally change my mind or learn something new and maybe gradually/subconsciously rethink my position.

Maybe I have too high bar for what counts as a bayesian updates - not sure?... (read more)

Replying toAnnouncing Trajectory Labs - A Toronto AI Safety Office

Ariel_9mo

Announcing Trajectory Labs - A Toronto AI Safety Office

Congrats! Can confirm that it is a great office :)

Replying toOrienting Toward Wizard Power

Ariel_9mo

Orienting Toward Wizard Power

my dream dwelling was a warehouse filled with whatever equipment one could possibly need to make things and run experiments in a dozen different domains

I had a similar dream, though I mostly thought about it from the context of "building cool fun mechanical stuff" and working on cars/welding bike frames. I think the actual usefulness might be a bit overrated, but still would be fun to have.

I do have a 3D printer though, and a welder (though I don't have anywhere to use it - needs high voltage plug). Again though not sure how useful these things are - it seems to me like it is mostly for fun, and in the... (read more)

Replying toThe EU Is Asking for Feedback on Frontier AI Regulation (Open to Global Experts)—This Post Breaks Down What’s at Stake for AI Safety

Ariel_10mo

The EU Is Asking for Feedback on Frontier AI Regulation (Open to Global Experts)—This Post Breaks Down What’s at Stake for AI Safety

Sure! and yeah regarding edits - I have not gone through the full request for feedback yet, I expect to have a better sense late next week of which contributions are most needed and how to prioritize. I mainly wanted to comment first on obvious things that stood out to me from the post.

There is also an Evals workshop in Brussels on Monday where we might learn more. I've know of some some non-EU based technical safety researchers who are attending, which is great to see.

With o1, and now o3, It seems fairly plausible now that there will be a split between "verifiable" capabilities, and general capabilities. Sure, there will be some cross-pollination (transfer), but this might have some limits.

What then? Can a superhuman mathematical + Coding AI also just reason through political strategy, or will it struggle and make errors/fallback on somewhat generic ideas in training data?

Can we get a "seed-AI style" consequentialist in some domains, while it fails to perform above human level in others? I'd like to believe reasoning would transfer (as it should be universal), but I dont think reasoning is sufficient for some more fuzzy domains - the model also needs good heuristics.

The AI Control agenda seems more promising now (for both this reason and some others).

Signal boost for the "username hiding" on homepage feature in settings - it seems cool, will see if it changes how I use LW.

I wonder also about a "hide karma by default". Though less sure if that will actually achieve the intended purpose, as karma can be a good filter when just skimming comments and not reading in detail.

LW Feature request/idea for a feature - In posts that have lots of in-text links to other posts, perhaps add an LLM 1-2 sentence (context informed) summary in the hover preview?

I assume that for someone who has been around the forum for many years, various posts are familiar enough that name-dropping them in a link is sufficient to give context. But If I have to click a link and read 4+ other posts as I am going through one post, perhaps the LW UI can fairly easily build in that feature.

(suggesting it as a features since it does seem like LW is a place that experiments with various features not too different from this - ofc, I can always ask for a LLM summary manually myself if I need to)

Ariels Shortform

Ariel_

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

EU AI Act passed Plenary vote, and X-risk was a main topic

Ariel_

(mildly rewritten version of my EA forum post)

The EU AI Act was originally proposed in 2020 as a very "EU regulates stuff first" kind of legislation, trying to make sure EU values are upheld (fairness, transparency, democracy, etc). Several revisions (and some lobbying) later, GPAI (general purpose AI) and foundation model language was added, and it started looking a little more X-risk friendly.

After some recent political uncertainty, it passed with a strong majority at the EU Plenary meeting.

I found it fascinating to watch the the live session (from June 13th, the vote was on the 14th), where the Act was discussed by various EU parties. A few things that stood out to... (read more)