x

LESSWRONG
LW

jacquesthibs — LessWrong

jacquesthibs

Top postsTop post

jacquesthibs

Message

I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I'd like to collaborate with.

Website: https://jacquesthibodeau.com

Twitter: https://twitter.com/JacquesThibs

GitHub: https://github.com/JayThibs

LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/

3196

Ω

102

15

433

5y

jacquesthibs

I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I'd like to collaborate with.

Website: https://jacquesthibodeau.com

Twitter: https://twitter.com/JacquesThibs

GitHub: https://github.com/JayThibs

LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/

Top postsTop post

But is it really in Rome? An investigation of the ROME model editing technique

Thanks to Andrei Alexandru, Joe Collman, Michael Einhorn, Kyle McDonell, Daniel Paleka, and Neel Nanda for feedback on drafts and/or conversations which led to useful insights for this work. In addition, thank you to both William Saunders and Alex Gray for exceptional mentorship throughout this project. The majority of this work was carried out this summer. Many people in the community were surprised when I mentioned some of the limitations of ROME (Rank-One Model Editing), so I figured it was worth it to write a post about it as well as other insights I gained from looking into the paper. Most tests were done with GPT-2, some were done with GPT-J. The ROME paper (Locating and Editing Factual Associations in GPT) has been one of the most influential papers in the prosaic alignment community. It has several important insights. The main findings are: 1. Factual associations such as “The Eiffel Tower is in Paris” seem to be stored in the MLPs of the early-middle layers of a GPT model. As the Tower token passes through the network, the MLPs of the early-middle layers will write information (e.g. the Eiffel Tower’s location) into the residual so that the model can later read that information to generate a token about that fact (e.g. Paris). 2. Editing/updating the MLP of a single layer for a given (subject, relationship, object) association allows the model to generate text with the updated fact when using new prompts/sentences that include the subject tokens. For example, editing “The Eiffel Tower is in Paris Rome” results in a model that outputs “The Eiffel Tower is right across from St Peter’s Basilica in Rome, Italy. “ In this post, I show that the ROME edit has many limitations: * The ROME edit doesn’t generalize in the way you might expect. It’s true that if the subject tokens you use for the edit are found in the prompt, it will try to generalize from the updated fact. However, it doesn’t “generalize” in the following ways: * It is not direction-agnosti

105Dec 30, 2022

What Makes an AI Startup "Net Positive" for Safety?

A descriptive, not prescriptive, overview of current AI Alignment Research

Research agenda: Supervising AIs improving AIs

Automating AI Safety: What we can do today

There have been multiple recent calls for the automation of AI safety and alignment research. There are likely many people who would like to contribute to this space, but would benefit from clear directions for how to do so. Stemming from a recent SPAR project and in light of limitations...

Jul 25, 2025•38

What Makes an AI Startup "Net Positive" for Safety?

In light of the recent news from Mechanize/Epoch and the community discussion it sparked, I'd like to open a conversation about a question some of us grapple with: What constitutes a net-positive AI startup from an AI safety perspective, and what steps can founders take to demonstrate goodwill and navigate...

Apr 18, 2025•82

How much I'm paying for AI productivity software (and the future of AI use)

This post is broken down into two parts: 1. Which AI productivity tools am I currently using (as an alignment researcher)? 2. Why does it currently feel hard to spend +$1000/month on AI to increase one's productivity drastically? Which AI productivity tools am I currently using? Let's get right to...

Oct 11, 2024•59

Shane Legg's necessary properties for every AGI Safety plan

I've been going through the FAR AI videos from the alignment workshop in December 2023. I'd like people to discuss their thoughts on Shane Legg's 'necessary properties' that every AGI safety plan needs to satisfy. The talk is only 5 minutes, give it a listen: Otherwise, here are some of...

May 1, 2024•58

AISC Project: Benchmarks for Stable Reflectivity

Apply to work on this project with me at AI Safety Camp 2024 before 1st December 2023. Summary Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane. There are two main ways AIs can get better:...

Nov 13, 2023•17

Research agenda: Supervising AIs improving AIs

[This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.] TL;DR Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane....

Apr 29, 2023•76

Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky

New article in Time Ideas by Eliezer Yudkowsky. Here’s some selected quotes. In reference to the letter that just came out (discussion here): > We are not going to bridge that gap in six months. > > It took more than 60 years between when the notion of Artificial Intelligence...

Mar 29, 2023•294

Load More (7/22)