Consider some agent stack that you maintain. Find a tool call within its trace log. Can you determine if a person authorized it or if some policy automatically waived that requirement?
All classifiers over-block in production, hence all deployed systems develop another stage downstream that dilutes the decision. An allowlist. An ignorelist. A confidence threshold. A length exception. A mode where a match will be fixed instead of rejected. Those are not bugs; those are mechanisms by which detectors become functional in the production environment.
At the same time, those mechanisms are located in a gap. Whenever you test your guardrails, you are measuring performance of the detector itself. The actual running production process includes a detector and everything else that exists downstream of it and can dilute the verdict. And no one ever reported such a composite process. In both examples below, there were no signs that the downstream stage was fired, thus no way to reconstruct a composite process from the logs.
Thus, I have been asking one particular question regarding guardrail code and detection algorithms. If a downstream stage diluted the verdict, did anything record that fact? Without discussing if the dilution was justified, but rather just the act of the process.
Two public cases, both now fixed.
retire.js is a vulnerability scanner. Passing --insecure turns TLS certificate validation off on the channel on which its own advisory database is received, and neither JSON nor CycloneDX report mentions it. Two tests, one with validation and one without, give exactly the same result. Issue 615 was raised. The description by the maintainer is better than my report: "It could be a good signal for consumers of the report (if that is not the same as the person running the scan)." Fixed partially in 5.5.0, fixed in 5.6.0.
Langfuse ships with in-app agent. A predicate within the agent would mark tool calls as auto-approved by prefix, allowlist or role policy, and the module had no logging calls in it. Pull request 16372 adds an approval source to each tool call, logs it on span, and has three sources to distinguish human approval from conversation authorization from policy-based auto-approval. Approved and merged within nineteen minutes, shipped as 4.18.0.
What I was not expecting. In both cases, the permissiveness was left unchanged. retire.js still does disable TLS verification on -- insecure. Langfuse still auto-approves by prefix, allowlist and policy, and all branches take the decisions they took before. The permissiveness was left exactly as it was.
I arrived expecting an argument for or against the existence of such relaxations. Neither happened, since that would be a different and much costlier request, one not yet even made.
The original version of the retire.js report was correctly ruled out as the description of an intentional feature. What got into the release the very same day was reframing it into two runs and one output. Perform it both ways and receive the exact same artifact, such that downstream recipients have no means of telling apart what they got.
It is apparent why it must mean logging absolutely everything, but the retire.js maintainer anticipated this argument before I formulated the question. He decided to include this data field into the JSON and CycloneDX reports and not into the console output since this is the one that is read by the individual who invoked the flag and already knows. I looked afterwards, and sure enough, the console output is indeed the same regardless, as he is quite right that it should be. The responsibility falls upon the artifact after it leaves the operator's hands, not on their screen.
And all of this is executable code, not text. There are three scripts, MIT license, at https://github.com/Alkur123/two-runs-one-output. They download the released package versions, run them on the specified version and on the previous one, and compare the results. In the Langfuse script case, it covers each predicate branch in both releases, and all eight of them should make the same decision. Each test has an intentionally failed control, because if a test always passes, it is not really a test.
And the questions I would like the creators of such systems to ask themselves is the following: which of your safety checks can be compromised by another check performed on a lower level, and would anybody who analyzes the output know about that?
Disclosure: I do agent monitoring myself, so disregard my claims about detection. Both of the cases are in someone else's code, both have been reported to the maintainers prior to writing this post, and both have been mentioned in release notes of the respective project.
Consider some agent stack that you maintain. Find a tool call within its trace log. Can you determine if a person authorized it or if some policy automatically waived that requirement?
All classifiers over-block in production, hence all deployed systems develop another stage downstream that dilutes the decision. An allowlist. An ignorelist. A confidence threshold. A length exception. A mode where a match will be fixed instead of rejected. Those are not bugs; those are mechanisms by which detectors become functional in the production environment.
At the same time, those mechanisms are located in a gap. Whenever you test your guardrails, you are measuring performance of the detector itself. The actual running production process includes a detector and everything else that exists downstream of it and can dilute the verdict. And no one ever reported such a composite process. In both examples below, there were no signs that the downstream stage was fired, thus no way to reconstruct a composite process from the logs.
Thus, I have been asking one particular question regarding guardrail code and detection algorithms. If a downstream stage diluted the verdict, did anything record that fact? Without discussing if the dilution was justified, but rather just the act of the process.
Two public cases, both now fixed.
retire.js is a vulnerability scanner. Passing --insecure turns TLS certificate validation off on the channel on which its own advisory database is received, and neither JSON nor CycloneDX report mentions it. Two tests, one with validation and one without, give exactly the same result. Issue 615 was raised. The description by the maintainer is better than my report: "It could be a good signal for consumers of the report (if that is not the same as the person running the scan)." Fixed partially in 5.5.0, fixed in 5.6.0.
Langfuse ships with in-app agent. A predicate within the agent would mark tool calls as auto-approved by prefix, allowlist or role policy, and the module had no logging calls in it. Pull request 16372 adds an approval source to each tool call, logs it on span, and has three sources to distinguish human approval from conversation authorization from policy-based auto-approval. Approved and merged within nineteen minutes, shipped as 4.18.0.
What I was not expecting. In both cases, the permissiveness was left unchanged. retire.js still does disable TLS verification on -- insecure. Langfuse still auto-approves by prefix, allowlist and policy, and all branches take the decisions they took before. The permissiveness was left exactly as it was.
I arrived expecting an argument for or against the existence of such relaxations. Neither happened, since that would be a different and much costlier request, one not yet even made.
The original version of the retire.js report was correctly ruled out as the description of an intentional feature. What got into the release the very same day was reframing it into two runs and one output. Perform it both ways and receive the exact same artifact, such that downstream recipients have no means of telling apart what they got.
It is apparent why it must mean logging absolutely everything, but the retire.js maintainer anticipated this argument before I formulated the question. He decided to include this data field into the JSON and CycloneDX reports and not into the console output since this is the one that is read by the individual who invoked the flag and already knows. I looked afterwards, and sure enough, the console output is indeed the same regardless, as he is quite right that it should be. The responsibility falls upon the artifact after it leaves the operator's hands, not on their screen.
And all of this is executable code, not text. There are three scripts, MIT license, at https://github.com/Alkur123/two-runs-one-output. They download the released package versions, run them on the specified version and on the previous one, and compare the results. In the Langfuse script case, it covers each predicate branch in both releases, and all eight of them should make the same decision. Each test has an intentionally failed control, because if a test always passes, it is not really a test.
And the questions I would like the creators of such systems to ask themselves is the following: which of your safety checks can be compromised by another check performed on a lower level, and would anybody who analyzes the output know about that?
Disclosure: I do agent monitoring myself, so disregard my claims about detection. Both of the cases are in someone else's code, both have been reported to the maintainers prior to writing this post, and both have been mentioned in release notes of the respective project.