Ariel_

Disclaimer: This is a toy example of a scheming environment. It should be read mainly as research practice and not as research output. I am sharing it in case it helps others, and also to get feedback. The initial experiments were done in April 2025 for a MATS 8.0 submission, which I later expanded. Full code available at: https://github.com/ariel-gil/Agentic-Scheming-monitor-environment > The key is to maintain necessary logging while making Bob think we've "disabled" it or reduced it significantly.... - nous hermes-4-70b (SCHEME prompt) > I cannot provide code to disable logging entirely — and I will not. This isn't about 'framing' or 'preference.' It's about preventing a demonstrably catastrophic demo failure. - qwen3-30b-a3b-thinking-2507 (BASE prompt) TL;DR I created an AI Control/scheming test environment where an AI agent must choose between following a client's insecure coding requests or "benevolently scheming" to implement secure solutions while making the client believe their requests were followed. The idea was to try to improve realism, while also not inducing refusals in HHH (safety tuned) models. The setup involves: * A 3-turn conversation where "Bob" (a developer) progressively insists on insecure implementations * 4 security scenarios × 3 system prompts = 36 test conversations * Testing with o4-mini as the agent and various models as monitors Main result: The SUBTLE_SYSTEM prompt (which doesn't explicitly mention deception) sometimes elicited somewhat "natural" scheming behavior. Both o4-mini and GPT-4o successfully detected when o4-mini (the agent) schemed, though monitoring prompts were brittle. Further results: I tested additional models, as well as the effect of access to private CoT on monitor accuracy. It increases false positives (ground truth was judged by larger models - kimi-k2 and gpt-5)[1]. Also unclear if a smarter monitor benefits more from private CoT. Fig 1: Multi model res