Maheep Chaudhary

Authors: Maheep Chaudhary, Ian Su, Nikhil Hooda, Nishith Shankar, Julian Tan, Kevin Zhu, Ryan Laggasse, Vasu Sharma, Ashwinee Panda Scatter plot with a smoothed trend line that shows AUROC absolute distance from 0.5 as a function of model size (billions of parameters, log scale). Each point shows the best-performing probe for a given model, with shapes and colors indicating various model families. Abstract Large language models (LLMs) can internally distinguish between evaluation and deployment contexts, a behaviour known as evaluation awareness. This raises significant concerns for AI safety, as models may behave differently between testing and real-world implementation. Prior work has demonstrated this behaviour in a single, large-scale open-source model (70B parameters), leaving room for investigation of whether other small or very large models exhibit the same capabilities. This project will investigate whether other language models also exhibit evaluation awareness and whether it scales predictably with model size. Through examining models of varying sizes from the Qwen and LLaMA families, we aim to identify whether this general trend will emerge. By showcasing how this behaviour varies across parameter sizes, it will contribute to our understanding of the deceptive, context-sensitive behaviours of LLMs and guide further research into designing more robust evaluation strategies for AI safety. > Our paper has been published in the proceedings of the NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models (ResponsibleFM). See our paper on arXiv here. Introduction Large language models (LLMs) can internally distinguish between evaluation and deployment contexts—a behavior known as evaluation awareness that fundamentally undermines AI safety evaluations Nguyen et al., 2025. When models recognize they are being tested, they may conceal dangerous capabilities during safety assessments or strat