Using fine-tuned models locally, for privacy concerns, can also be dangerous. They might be trained to perform well on a specific task, but do something secretly, which we won't have any reliable way to find out about.
Hugging Face runs its own scans, and Safetensors helps, but neither solves this problem, as these only check for dangerous autoruns on their load or for dangerous imports. But what if the model is fine-tuned in a way to prioritise outdated or vulnerable libraries when used as a coding assistant?
Therefore, a quick check is to run safety benchmarks on both the base and fine-tuned models, compare the risks, and if the gap is large, avoid using that fine-tuned model. This won't catch everything though, since a model can act normal on benchmarks and only do the hidden thing when it sees a specific trigger. Also, prefer downloading fine-tuned models from legitimate publishers.
I have been working with ADL (Activation Difference Lens, https://arxiv.org/abs/2510.13900), and trying to find out the behaviour of the model, as using the traces, we can already find out the topic it was fine-tuned on, at least for narrow fine-tunes.
Any suggestion on a better check than comparing benchmarks?

Research examples: Sleeper Agents (https://arxiv.org/abs/2401.05566), Sleeper Cell (https://arxiv.org/abs/2603.03371)