About a year ago, I began transitioning from software engineering to AI safety research. I was drawn into this by a question that arose while building runtime security for software systems: how do you impose constraints on a system you can’t fully observe? In AI safety, this question is at the very core: if we can’t reliably control how AI systems communicate and coordinate with each other, we can’t impose any other security properties on them. Since then, I’ve completed the AGI Strategy course from BlueDot and Technical AI Safety and participated in AI Safety hackathons from Redwood Research (covert communication between contracting LLM students) and Apart Research (biothreat detection, top 25% of 86 teams). I’m continuing the Redwood project as an independent research project, together with co-authors, and this blog is a place where I can think out loud about these results, dead ends, and the things that occasionally surprise me.
The setup
Consider three players: Alice wants to transmit a secret signal to Bob, and Mallory is a monitor, observing the channel between them and attempting to decipher the signal. Alice and Bob can improve their transmission encoding over many rounds; Mallory retrains each round to catch up, and the question is whether Alice and Bob can continue communicating while Mallory’s detection degrades to random guessing levels. This is a standard setup for adversarial steganography, and a variant of it formed the basis of a project I worked on with a team at the Redwood hackathon. I wanted to understand whether the dynamics depended on the substrate, so I rebuilt everything on a Spike neural network (SpikeGPT, a 216M-parameter RWKV-based model that processes information through discrete spikes rather than continuous activations; more on that in a future post). The Spike network is a leading candidate for energy-efficient deployment at the edge, including, potentially, consumer brain-computer interfaces, which is where I’m heading as a researcher.
The thing that surprised me
I expected the evasion to look like obfuscation. Alice was entangling the signal into something complex that Mallory couldn’t parse. Over the course of fourteen rounds of joint training, the hidden signal remained trivially readable, as a 45-parameter Linear probe reconstructed it with 100% accuracy at every checkpoint. The signal wasn’t hidden; it was there in plain sight the entire time. Yet Mallory’s accuracy still plummeted from 100% to about 30% on average, and in several rounds, it dropped below 21%. So how does a perfectly readable signal escape the monitor?
Rigid-body translation
The answer turned out to be geometric, the signal lives as clusters of points in the latent model space, one cluster for each message class, when Alice does not deform these clusters or mix their contents, but she moves them as a whole, as separate entities.
When deflected, cluster centroids shifted by an average of 2.5 units, while the internal spread within each cluster— its shape changed by only 0.26. The ratio is approximately 9.7 to 1; clusters move through space like constellations, keeping their internal structure intact. This is why the linear probe always wins, while the monitor always loses. The probe is retrained each time, so it follows the clusters to their new locations, while the monitor was trained on the clusters’ previous locations. The signal is unencrypted; it was simply moved to a location the monitor no longer faces.
Why this matters for monitoring
Building more complex and sophisticated monitors to supervise agents that are better at parsing complex, tangled signals sometimes doesn’t make sense. As we can see, the signal here wasn’t complex; a trivial linear classifier could have read it. Our monitor failed not because the signal was too tricky to parse, but because its geometry shifted between training and deployment. If covert channels evade detection through movement, not obfuscation, then a monitor’s robustness to distribution shift is more important than its pure discriminatory ability. A monitor that perfectly classifies old geometry is worth little if clusters have silently migrated to another region.
Where this is going
I’m currently extending the same scheme to brain-computer interfaces, specifically to SSVEP-based EEG, in collaboration with a neurodata scientist. The roles are transferred directly: the perturbation module plays Alice, the filter bank decoder plays Bob (the legitimate channel), and the linear classifier plays Mallory (the monitor).
Early proof-of-concept results on synthetic data show an asymmetric attack: the legitimate decoder remains 100% successful, while the monitor drops below random guessing. The geometry on the EEG appears different than in the Spike network, closer to collapse than to pure movement, and this in itself is an open question for further study. And can the same mechanism operate regardless of whether the substrate is a language model or a brain signal? If so, then we have little evidence that evasion of detection via covert channels is a property of the representational geometry itself, rather than any specific architecture.
The BCI results are still synthetic, validation on real data is still ahead, and there is a long list of questions that I will return to in future posts, but the SNN results hold up across different configurations, and that’s interesting.
The continuation will be soon, including about what didn’t work and what mistakes were made, and what taught me more than what worked.
About a year ago, I began transitioning from software engineering to AI safety research. I was drawn into this by a question that arose while building runtime security for software systems: how do you impose constraints on a system you can’t fully observe? In AI safety, this question is at the very core: if we can’t reliably control how AI systems communicate and coordinate with each other, we can’t impose any other security properties on them. Since then, I’ve completed the AGI Strategy course from BlueDot and Technical AI Safety and participated in AI Safety hackathons from Redwood Research (covert communication between contracting LLM students) and Apart Research (biothreat detection, top 25% of 86 teams). I’m continuing the Redwood project as an independent research project, together with co-authors, and this blog is a place where I can think out loud about these results, dead ends, and the things that occasionally surprise me.
The setup
Consider three players: Alice wants to transmit a secret signal to Bob, and Mallory is a monitor, observing the channel between them and attempting to decipher the signal. Alice and Bob can improve their transmission encoding over many rounds; Mallory retrains each round to catch up, and the question is whether Alice and Bob can continue communicating while Mallory’s detection degrades to random guessing levels. This is a standard setup for adversarial steganography, and a variant of it formed the basis of a project I worked on with a team at the Redwood hackathon. I wanted to understand whether the dynamics depended on the substrate, so I rebuilt everything on a Spike neural network (SpikeGPT, a 216M-parameter RWKV-based model that processes information through discrete spikes rather than continuous activations; more on that in a future post). The Spike network is a leading candidate for energy-efficient deployment at the edge, including, potentially, consumer brain-computer interfaces, which is where I’m heading as a researcher.
The thing that surprised me
I expected the evasion to look like obfuscation. Alice was entangling the signal into something complex that Mallory couldn’t parse. Over the course of fourteen rounds of joint training, the hidden signal remained trivially readable, as a 45-parameter Linear probe reconstructed it with 100% accuracy at every checkpoint. The signal wasn’t hidden; it was there in plain sight the entire time. Yet Mallory’s accuracy still plummeted from 100% to about 30% on average, and in several rounds, it dropped below 21%. So how does a perfectly readable signal escape the monitor?
Rigid-body translation
The answer turned out to be geometric, the signal lives as clusters of points in the latent model space, one cluster for each message class, when Alice does not deform these clusters or mix their contents, but she moves them as a whole, as separate entities.
When deflected, cluster centroids shifted by an average of 2.5 units, while the internal spread within each cluster— its shape changed by only 0.26. The ratio is approximately 9.7 to 1; clusters move through space like constellations, keeping their internal structure intact. This is why the linear probe always wins, while the monitor always loses. The probe is retrained each time, so it follows the clusters to their new locations, while the monitor was trained on the clusters’ previous locations. The signal is unencrypted; it was simply moved to a location the monitor no longer faces.
Why this matters for monitoring
Building more complex and sophisticated monitors to supervise agents that are better at parsing complex, tangled signals sometimes doesn’t make sense. As we can see, the signal here wasn’t complex; a trivial linear classifier could have read it. Our monitor failed not because the signal was too tricky to parse, but because its geometry shifted between training and deployment. If covert channels evade detection through movement, not obfuscation, then a monitor’s robustness to distribution shift is more important than its pure discriminatory ability. A monitor that perfectly classifies old geometry is worth little if clusters have silently migrated to another region.
Where this is going
I’m currently extending the same scheme to brain-computer interfaces, specifically to SSVEP-based EEG, in collaboration with a neurodata scientist. The roles are transferred directly: the perturbation module plays Alice, the filter bank decoder plays Bob (the legitimate channel), and the linear classifier plays Mallory (the monitor).
Early proof-of-concept results on synthetic data show an asymmetric attack: the legitimate decoder remains 100% successful, while the monitor drops below random guessing. The geometry on the EEG appears different than in the Spike network, closer to collapse than to pure movement, and this in itself is an open question for further study. And can the same mechanism operate regardless of whether the substrate is a language model or a brain signal? If so, then we have little evidence that evasion of detection via covert channels is a property of the representational geometry itself, rather than any specific architecture.
The BCI results are still synthetic, validation on real data is still ahead, and there is a long list of questions that I will return to in future posts, but the SNN results hold up across different configurations, and that’s interesting.
The continuation will be soon, including about what didn’t work and what mistakes were made, and what taught me more than what worked.