This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Disclosure: the experiments, results and analysis are mine; this write-up was drafted with Claude from my notes and results, and edited by me. The retraction chain below includes several errors Claude caught and several it made.
I spent a few weeks and about thirteen dollars of GPU time trying to test an idea about continual learning. The idea did not get tested. What I got instead was a methods finding and a structural result about why the experiment I designed could never have worked. Both seem worth writing down, and the sequence of wrong headlines that got me there seems worth writing down too, since I suspect the specific errors are common and mostly invisible.
What I was trying to do
A language model reads a skill description in context, works out what it implies, uses it, and then discards all of that at the end of the forward pass. Weights never change. The inference happened and nothing kept it. So the question was whether you could take the internal state produced by reading the skill and write it into weights — a write path from activation to weight — while penalising change in directions that don't carry the skill.
Anthropic's global workspace paper gave a convenient coordinate system for the target: a corpus-averaged Jacobian from a middle-layer activation to vocabulary logits, isolating a low-dimensional subspace that drives what the model reports. I used their released lens, validated it on Qwen2.5-7B and 14B, built a corpus of artificial procedural skills with verifiable pass/fail, and started measuring.
None of what follows is a criticism of that paper. Their interventions are lens-coordinate patches within a single forward pass — same prompt, same positions. The failure mode below is specific to a thing I was doing that they were not: copying activations from one prompt into a different prompt.
The retraction chain
"Injection response is linear, R² = 0.9997." Inject alpha times v into the residual stream, then measure the projection onto v. That is the baseline projection plus alpha times the inner product of v with the readout direction. Linear by construction. The R² was measuring my own arithmetic.
"Degraded readout produces a threshold." True by definition. Thresholding a monotonic signal produces a threshold, and the steepness is set by trial variance. This is signal detection theory, not a finding about the model.
"The text transitions sharply while the underlying projection stays linear." Argmax is a threshold. Same artefact in different clothes — the text change sat exactly where the logit derivative peaked.
"Concept injection reaches 100% at alpha = 13, on 60 of 60 prompts." It reached 100% because the model was emitting the same word over and over, at a repeat fraction of 97%. The sweep ran far past the point where the model still produces coherent text, and the scorer counted degenerate repetition as success. What survives after gating on a repeat-fraction ceiling is a rise from 1.7% to 86.7% over a much narrower range — a real effect, and a much smaller claim.
"Confinement loses to ordinary fine-tuning." Three independently disqualifying defects: the baseline arm got eight iterations of bisection search and the experimental arm got one hand-computed learning rate; install success was unmatched at 75% against 100%, so the two arms spent their weight-drift budget on different things; and it was one skill at one magnitude when the design calls for a slope across magnitudes.
Every one of those objections was available in data I already had. None required a new measurement. They surfaced when a reviewer with no investment in the result looked at it, which is the only part of this process I would defend as working.
The methods finding
To test whether the workspace state was a sufficient target, I ran the cheap version first: skip training, patch the captured with-file activations directly into a no-file run at inference, and see whether the skill appears. If a perfect copy doesn't install the behaviour, no weight update onto that target will either.
The two runs have different token counts — one has the skill spec in the prompt, one doesn't. So you need an alignment rule. I aligned by taking the shorter length and copying that many positions from the start.
That is wrong, and it is wrong in the worst possible direction.
On this corpus the no-file run had 58.46 positions on average, while the genuinely shared suffix was 33.46. So the naive alignment copied 25 positions that were not shared — and those positions contained the tail of the skill spec, including its worked example. The stored diagnostic shows the copied text ending in the example's answer token, followed by the instruction to answer with that token alone on a line beginning ANSWER.
That is the answer and the answer format, being copied in as activations and then scored as though the model had produced them.
On the one skill for which I have stored artefacts rather than terminal output, patching all 28 layers: the naive alignment gives 54.2% install, and correct token-suffix alignment gives 0.0%. The phantom rate varies with how much of the answer template happens to land in the copied tail.
I made this error four separate times, in four different scripts, because each one re-derived its own alignment instead of importing a shared function. If you are doing this kind of experiment, make the alignment a single audited function and make every script import it.
Why the experiment could not have worked
Correcting the alignment gave 0% install at every layer, every extent, every rank truncation, every noise level. A perfect copy of the entire activation tensor at every genuinely shared position transferred nothing. That looked like a strong negative result: whatever represents a skill, it isn't a state at those positions.
Then the positive control failed.
France to China is a known-good item edit — patch the workspace and the model says Beijing instead of Paris, and it replicates cleanly in other people's hands. Run through my identical token-aligned pipeline, with the readout sitting at the patch boundary, it succeeded on all four trials at all layers and one of four in the middle band. Move the readout a few tokens past the boundary — which is what the skill harness does, since the answer comes after some generated text — and it drops to zero of four in both conditions. The known-good edit fails exactly as the skill does.
The all-layers success was tautological. It includes the final layer, whose output feeds the unembedding, and the shared suffix always includes the final position — so it overwrote the next-token distribution and then read the next token. The giveaway was that it worked with as little as one shared position, and that position was the readout position. The only non-degenerate arm is the middle band: one of four against zero of twenty-four, Fisher exact p = 0.14.
The structural reason is simple in hindsight. The differing content sits in the non-shared prefix. "France" appears before the shared suffix, just as the skill spec appears before the shared task text. Token-aligned patching overwrites only the suffix and leaves the prefix key-value cache intact, so the model goes on reading "France" out of its own cache. The test is structurally incapable of transferring anything that lives in the prefix — which, in both experiments, is the entire thing of interest.
This is not a bug I can fix with a better alignment function. It rules out a family of designs: any experiment that compares a with-context run to a without-context run by copying activations between them will hit it, because the whole point is that one run has tokens the other lacks. There is no position to copy them into.
What survives
The alignment artefact, as above, and the prefix-geometry constraint — which is the useful part, since it tells you what a valid design has to avoid.
The 0% results are withdrawn as evidence about skills. They are the same structural artefact as the failed control.
One number is genuinely unaffected: the skill's activation change is only 1.23 times enriched in the workspace subspace over chance. About 5.6% of the squared difference falls inside a subspace that is roughly 6% of the variance. That is pure statistics on captured activations, with no patching involved, and it says a workspace-only target was always going to underdetermine the write.
Ordinary fine-tuning installs these skills robustly across the drift budget: 75% install at a weight drift of 0.083, rising through 83.3% at 0.098 and 91.7% at 0.202 and 0.367, to 100% at 0.883. So the skill is reachable in weight space by conventional means. Whether it can be reached selectively — without collateral damage elsewhere — is untouched by any of this.
The idea I set out to test is not refuted. It is untested, and I now know one class of experiment that cannot test it.
The part I would generalise
Four of the five retractions above are one error: I built the measurement, then asked what it measured. Optimise onto a coordinate, hit the coordinate, discover the coordinate wasn't the thing.
The rule that would have caught all of them, and which I now apply before writing any measurement code: write down what the number is under the null and under the alternative, and refuse to write the code if they are the same. A monotonic signal plus any threshold produces a step. Argmax is a threshold. Rank is a threshold. Binarised report is a threshold. If your measurement is a threshold and your hypothesis is about thresholds, you will find what you are looking for regardless of whether it is there.
The second thing that worked: every headline got handed to a fresh context with no framing and the instruction to find the reason it was wrong, before it went anywhere. That is what caught the tautological control, and it caught it about an hour before this post would have been wrong in public.
Disclosure: the experiments, results and analysis are mine; this write-up was drafted with Claude from my notes and results, and edited by me. The retraction chain below includes several errors Claude caught and several it made.
I spent a few weeks and about thirteen dollars of GPU time trying to test an idea about continual learning. The idea did not get tested. What I got instead was a methods finding and a structural result about why the experiment I designed could never have worked. Both seem worth writing down, and the sequence of wrong headlines that got me there seems worth writing down too, since I suspect the specific errors are common and mostly invisible.
What I was trying to do
A language model reads a skill description in context, works out what it implies, uses it, and then discards all of that at the end of the forward pass. Weights never change. The inference happened and nothing kept it. So the question was whether you could take the internal state produced by reading the skill and write it into weights — a write path from activation to weight — while penalising change in directions that don't carry the skill.
Anthropic's global workspace paper gave a convenient coordinate system for the target: a corpus-averaged Jacobian from a middle-layer activation to vocabulary logits, isolating a low-dimensional subspace that drives what the model reports. I used their released lens, validated it on Qwen2.5-7B and 14B, built a corpus of artificial procedural skills with verifiable pass/fail, and started measuring.
None of what follows is a criticism of that paper. Their interventions are lens-coordinate patches within a single forward pass — same prompt, same positions. The failure mode below is specific to a thing I was doing that they were not: copying activations from one prompt into a different prompt.
The retraction chain
"Injection response is linear, R² = 0.9997." Inject alpha times v into the residual stream, then measure the projection onto v. That is the baseline projection plus alpha times the inner product of v with the readout direction. Linear by construction. The R² was measuring my own arithmetic.
"Degraded readout produces a threshold." True by definition. Thresholding a monotonic signal produces a threshold, and the steepness is set by trial variance. This is signal detection theory, not a finding about the model.
"The text transitions sharply while the underlying projection stays linear." Argmax is a threshold. Same artefact in different clothes — the text change sat exactly where the logit derivative peaked.
"Concept injection reaches 100% at alpha = 13, on 60 of 60 prompts." It reached 100% because the model was emitting the same word over and over, at a repeat fraction of 97%. The sweep ran far past the point where the model still produces coherent text, and the scorer counted degenerate repetition as success. What survives after gating on a repeat-fraction ceiling is a rise from 1.7% to 86.7% over a much narrower range — a real effect, and a much smaller claim.
"Confinement loses to ordinary fine-tuning." Three independently disqualifying defects: the baseline arm got eight iterations of bisection search and the experimental arm got one hand-computed learning rate; install success was unmatched at 75% against 100%, so the two arms spent their weight-drift budget on different things; and it was one skill at one magnitude when the design calls for a slope across magnitudes.
Every one of those objections was available in data I already had. None required a new measurement. They surfaced when a reviewer with no investment in the result looked at it, which is the only part of this process I would defend as working.
The methods finding
To test whether the workspace state was a sufficient target, I ran the cheap version first: skip training, patch the captured with-file activations directly into a no-file run at inference, and see whether the skill appears. If a perfect copy doesn't install the behaviour, no weight update onto that target will either.
The two runs have different token counts — one has the skill spec in the prompt, one doesn't. So you need an alignment rule. I aligned by taking the shorter length and copying that many positions from the start.
That is wrong, and it is wrong in the worst possible direction.
On this corpus the no-file run had 58.46 positions on average, while the genuinely shared suffix was 33.46. So the naive alignment copied 25 positions that were not shared — and those positions contained the tail of the skill spec, including its worked example. The stored diagnostic shows the copied text ending in the example's answer token, followed by the instruction to answer with that token alone on a line beginning ANSWER.
That is the answer and the answer format, being copied in as activations and then scored as though the model had produced them.
On the one skill for which I have stored artefacts rather than terminal output, patching all 28 layers: the naive alignment gives 54.2% install, and correct token-suffix alignment gives 0.0%. The phantom rate varies with how much of the answer template happens to land in the copied tail.
I made this error four separate times, in four different scripts, because each one re-derived its own alignment instead of importing a shared function. If you are doing this kind of experiment, make the alignment a single audited function and make every script import it.
Why the experiment could not have worked
Correcting the alignment gave 0% install at every layer, every extent, every rank truncation, every noise level. A perfect copy of the entire activation tensor at every genuinely shared position transferred nothing. That looked like a strong negative result: whatever represents a skill, it isn't a state at those positions.
Then the positive control failed.
France to China is a known-good item edit — patch the workspace and the model says Beijing instead of Paris, and it replicates cleanly in other people's hands. Run through my identical token-aligned pipeline, with the readout sitting at the patch boundary, it succeeded on all four trials at all layers and one of four in the middle band. Move the readout a few tokens past the boundary — which is what the skill harness does, since the answer comes after some generated text — and it drops to zero of four in both conditions. The known-good edit fails exactly as the skill does.
The all-layers success was tautological. It includes the final layer, whose output feeds the unembedding, and the shared suffix always includes the final position — so it overwrote the next-token distribution and then read the next token. The giveaway was that it worked with as little as one shared position, and that position was the readout position. The only non-degenerate arm is the middle band: one of four against zero of twenty-four, Fisher exact p = 0.14.
The structural reason is simple in hindsight. The differing content sits in the non-shared prefix. "France" appears before the shared suffix, just as the skill spec appears before the shared task text. Token-aligned patching overwrites only the suffix and leaves the prefix key-value cache intact, so the model goes on reading "France" out of its own cache. The test is structurally incapable of transferring anything that lives in the prefix — which, in both experiments, is the entire thing of interest.
This is not a bug I can fix with a better alignment function. It rules out a family of designs: any experiment that compares a with-context run to a without-context run by copying activations between them will hit it, because the whole point is that one run has tokens the other lacks. There is no position to copy them into.
What survives
The alignment artefact, as above, and the prefix-geometry constraint — which is the useful part, since it tells you what a valid design has to avoid.
The 0% results are withdrawn as evidence about skills. They are the same structural artefact as the failed control.
One number is genuinely unaffected: the skill's activation change is only 1.23 times enriched in the workspace subspace over chance. About 5.6% of the squared difference falls inside a subspace that is roughly 6% of the variance. That is pure statistics on captured activations, with no patching involved, and it says a workspace-only target was always going to underdetermine the write.
Ordinary fine-tuning installs these skills robustly across the drift budget: 75% install at a weight drift of 0.083, rising through 83.3% at 0.098 and 91.7% at 0.202 and 0.367, to 100% at 0.883. So the skill is reachable in weight space by conventional means. Whether it can be reached selectively — without collateral damage elsewhere — is untouched by any of this.
The idea I set out to test is not refuted. It is untested, and I now know one class of experiment that cannot test it.
The part I would generalise
Four of the five retractions above are one error: I built the measurement, then asked what it measured. Optimise onto a coordinate, hit the coordinate, discover the coordinate wasn't the thing.
The rule that would have caught all of them, and which I now apply before writing any measurement code: write down what the number is under the null and under the alternative, and refuse to write the code if they are the same. A monotonic signal plus any threshold produces a step. Argmax is a threshold. Rank is a threshold. Binarised report is a threshold. If your measurement is a threshold and your hypothesis is about thresholds, you will find what you are looking for regardless of whether it is there.
The second thing that worked: every headline got handed to a fresh context with no framing and the instruction to find the reason it was wrong, before it went anywhere. That is what caught the tautological control, and it caught it about an hour before this post would have been wrong in public.