This is a brief summary of what we believe to be the most important takeaways from our new paper and from our findings shown in the o1 system card. We also specifically clarify what we think we did NOT show. 

Paper: https://www.apolloresearch.ai/research/scheming-reasoning-evaluations 

Twitter about paper: https://x.com/apolloaisafety/status/1864735819207995716 

Twitter about o1 system card: https://x.com/apolloaisafety/status/1864737158226928124 

What we think the most important findings are

Models are now capable enough to do in-context scheming reasoning

We say an AI system is “scheming” if it covertly pursues misaligned goals, hiding its true capabilities and objectives. We think that in order to scheme, models likely need to be goal-directed, situationally aware, and capable enough to reason about scheming as a strategy. In principle, models might acquire situational awareness and stable long-term goals during training, and then scheme in pursuit of those goals. We...

The ACX/EA/LW Sofia Meetup for October will be on the 15th (Sunday) at 17:00 at the Mr. Pizza on Vasil Levski.

Sofia ACX started with the 2021 Meetups Everywhere round. Attendance hovers around 4-8 people. Everyone worries they're not serious enough about ACX to join, so you should banish that thought and come anyway.  "Please feel free to come even if you feel awkward about it, even if you’re not 'the typical ACX reader', even if you’re worried people won’t like you", even if you didn't come to the previous meetings, even if you don't speak Bulgarian, etc., etc.

Each month we pick something new to read and discuss. This time, we're discussing "Math's Fundamental Flaw," a Veritasium video on Gödel's Incompleteness Theorem.  

See you there.

Epistemic status: Toy model. Oversimplified, but has been anecdotally useful to at least a couple people, and I like it as a metaphor.

Introduction

I’d like to share a toy model of willpower: your psyche’s conscious verbal planner “earns” willpower (earns trust with the rest of your psyche) by choosing actions that nourish your fundamental, bottom-up processes in the long run.  For example, your verbal planner might expend willpower dragging you to disappointing first dates, then regain that willpower, and more, upon finding you a good long-term romance.  Wise verbal planners can acquire large willpower budgets by making plans that, on average, nourish your fundamental processes.  Delusional or uncaring verbal planners, on the other hand, usually become “burned out” – their willpower budget goes broke-ish, leaving them little to...

My paper with my Ph.D. advisor Vince Conitzer titled "Extracting Money from Causal Decision Theorists" has been formally published (Open Access) in The Philosophical Quarterly. Probably many of you have seen either earlier drafts of this paper or similar arguments that others have independently given on this forum (e.g., Stuart Armstrong posted about an almost identical scenario; Abram Demski's post on Dutch-Booking CDT also has some similar ideas) and elsewhere (e.g., Spencer (forthcoming)  and Ahmed (unpublished) both make arguments that resemble some points from our paper).

Our paper focuses on the following simple scenario which can be used to, you guessed it, extract money from causal decision theorists:

Adversarial Offer: Two boxes, $B_1$ and $B_2$, are on offer. A (risk-neutral) buyer may purchase one or none of the boxes but not both.

TL:DR: Recently, Lucius held a presentation on the nature of deep learning and why it can generalise to new data. Kaarel, Dmitry and Lucius talked about the slides for that presentation in a group chat. The conversation quickly became a broader discussion on the nature of intelligence and how much we do or don't know about it. 

Background 

Lucius:  I recently held a small talk presenting an idea for how and why deep learning generalises. It tried to reduce concepts from Singular Learning theory back to basic algorithmic information theory to sketch a unified picture that starts with Solomonoff induction and, with a lot of hand waving, derives that under some assumptions, just fitting a big function to your data using a local optimisation method like stochastic gradient descent...

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.

Previously: Sadly, FTX

I doubted whether it would be a good use of time to read Michael Lewis’s new book Going Infinite about Sam Bankman-Fried (hereafter SBF or Sam). What would I learn that I did not already know? Was Michael Lewis so far in the tank of SBF that the book was filled with nonsense and not to be trusted?

I set up a prediction market, which somehow attracted over a hundred traders. Opinions were mixed. That, combined with Matt Levine clearly reporting having fun, felt good enough to give the book a try.

I need not have worried.

Going Infinite is awesome. I would have been happy with my decision on the basis of any one of the following:

The details I learned or clarified about the psychology of SBF...