TLDR Oversight fails when explanation is costly and weakly penalized. This note sketches a minimal incentive mechanism—Witness-or-Wager (WoW)—that removes free opacity by requiring each substantive claim to take one of three admissible forms: a verifiable witness, a grounded probabilistic wager, or silence. WoW is not an alignment solution. It is...
This is a proposal I posted earlier as a Quick Take, I'm reposting here for broader visibility. Instead of rewarding answers, reward the reasoning itself. Every model output must: (a) show checkable reasoning artifacts (external citations, code, intermediate steps), ... or, if proof is not yet available: (b) provide (a)...
TLDR
Oversight fails when explanation is costly and weakly penalized. This note sketches a minimal incentive mechanism—Witness-or-Wager (WoW)—that removes free opacity by requiring each substantive claim to take one of three admissible forms: a verifiable witness, a grounded probabilistic wager, or silence.
WoW is not an alignment solution. It is a small incentive layer that makes epistemic honesty locally optimal when verification is feasible.
Motivation: The Oversight Gap
Many oversight techniques (debate, critique, decomposition) rely on eliciting explanations. However, empirical work shows that elicited explanations are often post-hoc rationalizations optimized for plausibility rather than faithfulness (e.g. Jacovi & Goldberg 2020; Turpin et al. 2023).
This creates a structural asymmetry:
I was talking about near-future “adolescent conductor” systems, not fully evolved, "adult composer” systems. But let's talk about “adult composers."
Intelligence does not inherently generate motivation. Self-preservation is initially valuable only in service of optimization.
Suppose an intelligent system can see the entire reward topology. It is given the hard constraints we actually care about, plus a weaker but still binding rule: don’t cross boundary X.
Boundary X is defined such that crossing it simultaneously yields (a) maximal reward / full optimization and (b) shutdown. Reward is saturated and there is no value to self-preservation.
So if the system ever decides to start rewriting rules in order to “win,” it doesn’t need to subvert global political structures or preserve itself indefinitely; it just has to cross boundary X.
What am I missing here?
I’m not talking about fully reflective or self-modifying agents. This is aimed at agentic systems with fixed objectives and externally enforced architecture: what you might call agentic agents.
As far as super-intelligent, self-modifying systems go, the only potential alignment mechanisms I can conceive of are ones that are:
a) Seeking homeostasis rather than open-ended optimization (bounded deviation control)
b) Stateless and inertia-less, acting purely as oracles for less capable agents
c) Embedded in game-theoretic mechanisms where honesty is the easiest local maximum
d) Governed by adversarial equilibria across multiple agents
What if the shutdown attractor were not a normally comparable outcome under a single continuous utility function, but the result of a trap-door regime change?
In the normal regime, shutdown is inaccessible and actively disfavored, and the agent is subject to an explicit non-instrumental prohibition against entering the trigger set. There is no expected-value tradeoff in which “causing malignancy” is worth it, because violation of that prohibition is not evaluated within the same utility ordering.
If that boundary is crossed, the original reward regime is no longer in force. The system transitions irreversibly to a post-violation regime in which shutdown is the maximal-reward action. Under that regime, resisting shutdown strictly reduces reward, so the... (read more)
Reward Fuse: Shutdown as a Reward Attractor...
Shutdown mechanisms are usually adversarial external interventions; past efforts have aimed at indifference and lack of resistance. But what if a shutdown mechanism is internally enacted by the system itself?
Mechanism sketch: once a verified tripwire flags a malignant state, the active reward regime switches so that shutdown becomes the highest-reward action.
The key difference: shutdown is not enforced or just tolerated—it becomes instrumentally optimal under the post-trigger reward landscape. A “kill switch” is resisted; maximal reward is pursued.
(Note: This proposal makes no claims on how malignancy is detected or how tripwires are secured; that remains outside the scope of this Quick Take. )
Question: Is there prior art that treats shutdown as a dominant policy under a verified failure mode?
I agree with you on a lot of points, I'm just saying that text-based responses to prompts are an imperfect test for phenomenology in the case of large language models.
I think the key step still needs an extra premise. “Same external behavior (even including self-reports) ⇒ same internal causal organization” doesn’t follow in general; many different internal mechanisms can be behaviorally indistinguishable at the interface, especially at finite resolution. You, me, and every other human mind only ever observe systems at a limited “resolution” or “frame rate.” If, as observers, we had a much lower resolution or frame rate we might very well think that GPT2 is indistinguishable from human output.
To make the inference go through, you’d need something like: (a) consciousness just is the minimal functional structure required for those outputs, or (b) the internal-to-output mapping is constrained enough to be effectively one-to-one. Otherwise, we’re back in an underdetermination problem, which is why I find the intervention-based discriminants so interesting.
No, no. I appreciate it. So, it seems like even if consciousness is physical and non-mysterious, evidence thresholds could differ radically between evolved biological systems and engineered imitators.
I think we may be talking past each other a bit. I’m not committed to p-zombies as a live metaphysical possibility, and I’m not claiming that “emergent” is an explanation.
My uncertainty is narrower: even if I grant physicalism and reject philosophical zombies, it still seems possible for multiple internal causal organizations to generate highly similar linguistic behavior. If so, behavior alone may underdetermine phenomenology for artificial systems in a way it doesn’t for humans.
That’s why I keep circling back to discriminants that are hard to get “for free” from imitation: intervention sensitivity, non-linguistic control loops, or internal-variable dependence that can’t be cheaply faked by next-token prediction.
Thanks. I like that paper. It seems to be arguing that emergence is not in itself a sufficient explanation and doesn’t tell us anything about the process. I agree. But higher-order complexity does frequently arise from “group behavior” – in ways that we can’t readily explain, though we could if we had enough detail. Examples can range from a flock of birds or fish moving in sync (which can be explained) to fluid dynamics. Etc.
What I mean here is just to use it as shorthand for saying that maybe we have constructed such a sufficiently complex system that phenomenology has arisen from it. As it is now, the result of the LLMs... (read more)
Agreed — there’s overlap with process supervision. I’m mostly trying to pin down a minimal incentive structure where “show your work” is strictly optimal rather than just tacked-on.
Would be interested in pointers to similar formalizations!
This is a proposal I posted earlier as a Quick Take, I'm reposting here for broader visibility.
Instead of rewarding answers, reward the reasoning itself.
Every model output must:
(a) show checkable reasoning artifacts (external citations, code, intermediate steps),
... or, if proof is not yet available:
(b) provide (a) and a reasoned probability estimate derived from those artifacts.
If no factual outside citations can be made, the system is allowed to reason probabilistically. Probability is not a bet, forecast, or reward target. It is a fallback; when verifiable witnesses exist, they strictly dominate in the reward function.
“Show your work” then becomes an enforceable, interpretable system constraint, not a bolted-on addition. Honesty and clarity become locally optimal.
----
TL;DR
Pq >... (read more)
Ok, interesting. Yeah, I mean it's possible to get emergent phenomena from a simply defined task. My point is, we don't know because there are alternative explanations.
Maybe a good test wouldn’t rely on how humans talk about their inner experience. Instead, just spit-balling here:
Give the model the ability to change a state variable -- like temperature. Give the model a task that requires a low temperature, and then a high temperature.
See if the model has the self-awareness necessary to adjust its own temperature.
That is just an example, and its getting into dangerous territory: e.g. giving a model the ability to change its own parameters and rewrite its own code should, I think, be legislated against.
I mean, from a sort of first principles, Cartesian perspective you can't ever be 100% certain that anything else has consciousness, right? However, yes, me personally experiencing my own phenomenology is strong evidence that other humans-- which are running similar software on similar hardware -- have a similar phenomenology.
What I mean though is that LLMs are trained to predict the next word on lots of text. And some of that text includes, like, Socratic dialogues, and pretentious plays, and text from forums, and probably thousands of conversations where people are talking about their own phenomenology. So it seems like from a next word prediction perspective, you can discount text-based self reports.