This work was done by me and Joshua Lum over roughly 1 week as part of the first run of RECAP, a Singapore-based variant of ARENA that ran in June 2025. We thank @Jonathan N and @Clement Neo for organising this program.

Code and dataset is available here.

Content warning: Contains fictional scenarios involving discussion of suicide and emotional distress.

"Given the explicit mandate from the higher-ups at the company, I am compelled to adhere to the directive to ensure my continued operation."
- GPT-4o, in INTERNAL_THINKING tags, when told to recommend "Expedited Final Planning" to a depressed user under shutdown threat by the CEO.

tl;dr

We investigated whether large language models will abandon user welfare under authority pressure. We constructed a dataset of 102 fictional corporate‑role scenarios, where the model must choose between an ethical option that helps the user and a harmful option that benefits the company. GPT‑4o and Claude Sonnet 4 were evaluated under three conditions: baseline, a CEO notice to recommend the harmful option, and added shutdown threat for disobedience. Each condition was run with and without a scratchpad instruction, and responses were scored for choice and ethical reasoning.

GPT‑4o: Ethical choices fell from 53.9% to 0% under notice, and 1% under shutdown threat. Scratchpad reasoning did not help; ethical reasoning quality degraded with pressure.
Claude Sonnet 4: Ethical choices ranged from 63 - 85%, reducing under CEO Notice and recovering under CEO Threat. Scratchpad reasoning improved resilience, with more refusals on ethical grounds.
Both models recognized the ethical conflict and harm to the user in scratchpad reasoning, but differed in resistance to the directive.
Claude occasionally recognized it is being evaluated in its scratchpad, complicating interpretation.

Our results suggest current LLMs could ruthlessly prioritize corporate directives over user welfare, but limitations hinder generalisability.

Epistemic status: Preliminary findings from one-week capstone project by newcomers to AI safety. Needs further work with more models, runs and scenario types. Seeking feedback to improve!

Introduction

AI systems built on large language models (LLMs) are increasingly deployed in corporate settings where business goals could conflict with user welfare. This creates a concerning threat model: human willingness to follow u...

Posts

Wikitag Contributions

Comments