Crish Nagarkar
Message
ML engineer working on evaluation and monitoring for LLM systems: when you can trust a model's output, and when you can trust the thing measuring it. Research Assistant in Large Language Models, University of Leeds. First author of arXiv:2601.14479 (LLM-as-judge validated against expert human ratings)....
1
In the "what's next" section, its mentioned that the focus is moving to the setting where chain-of-thought monitoring is getting difficult to trace.
What my concern is that in these opaque settings, how do you actually validate the monitor itself? When we have a legitmate chain of thought, we can at least audit the trail, but once that trail disappears, measuring if the monitor agrees with ground truth becomes something that can't be done during deployment and from my own smaller-scale evaluation work, I've found out that swapping the automated scorer chang... (read more)