Thx, I think I got most of this from your top level comment & Mikhail's post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I'd disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.

My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there'd have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that's worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.

I expect you'll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there's way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.

I don't know how costly/beneficial this screw up concretely was to humanity's survival, but I guess that total cost would've been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.

Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don't have a way to cheaply fix that mistake (which would bound the risk).

And there isn't an equally strong opposing force afaict? I can imagine blabbering out something that I'd afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would've been better to allow him to share it.

Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn't ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn't expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.

Disclosure: Written by ChatGPT on 2025-09-16 at my request.

Energy use in isometric holds

Abstract. Standard physiology; no novelty. When an object is held stationary, static equilibrium ($\sum F=0$, $\sum \tau =0$) sets the joint torque needed to counter gravity. Muscle fibers generate that torque. Force in skeletal muscle is produced by continuous actomyosin cross-bridge cycling; each cycle uses one ATP. Calcium must be kept high to permit attachment and then pumped back into the sarcoplasmic reticulum; the Na${}^{+}$/K${}^{+}$ pump maintains excitability. External mechanical work is approximately zero, but chemical energy is consumed continuously and dissipated as heat. Example: holding a $5kg$ bag at the hand (forearm $r\approx 0.35m$, elbow ${90}^{\circ }$) requires $T\approx 17Nm$. Relative to typical maximum elbow-flexion torque ($\approx 70Nm$ in young men, $\approx 35Nm$ in young women), this corresponds to $\approx 24$–$49\%$ MVC. Forearm $\dot{V}{O}_2$ in that range is $\approx 20$–$30mLmi{n}^{-1}$, i.e., $\approx 7$–$10W$ of local heat; two minutes dissipates $\approx 0.8$–$1.2kJ$.

Model

1. Static equilibrium (mechanics). Gravity exerts a force $mg$ at a horizontal distance $r$ from the elbow. The required elbow torque is $T_{req}\approx mgr$ (simple forearm model).
2. Torque to muscle force (anatomy). Elbow flexors act through an effective moment arm $r_{eff}$. The muscle force satisfies $T\approx F_m{r}_{eff}$. Holding a constant torque implies a near-constant average $F_m$ (ignoring co-contraction and passive tissues for this shortform).
3. Force generation (molecular). Each muscle fiber contains many myosin heads that attach to actin briefly, pull, and detach. Only a fraction are attached at any instant, so maintaining force requires continuous replacement of attached heads; each replacement hydrolyzes one ATP. Attachment requires elevated Ca${}^{2+}$; relaxation requires pumping Ca${}^{2+}$ back into the sarcoplasmic reticulum.
4. Energy balance (physiology). External work is $\approx 0$ at zero velocity, but internal processes consume ATP: $P_{ATP}=P_{xb}+P_{act}$. Reviews place activation at roughly 25–45% of ATP turnover in isometric contractions (muscle-dependent). At higher %MVC and longer holds, intramuscular pressure restricts blood flow, limiting oxygen delivery and accelerating fatigue.^[1][2][3]

Worked example

Setup. $m=5kg$; $r\approx 0.35m$; elbow ${90}^{\circ }$.

• Torque required: $T\approx mgr=5\times 9.81\times 0.35\approx 17Nm$.
• Relative intensity: $MVC\approx 70Nm$ (young men, $\sim {120}^{\circ }$) or $35Nm$ (young women, $\sim {90}^{\circ }$) $\Rightarrow$ $24\%$ or $49\%$ MVC.^[4]
• Metabolic cost (from oxygen): around 20–30% MVC, forearm $\dot{V}{O}_2\approx 20$–$30mLmi{n}^{-1}$. With $\approx 20.9kJ{L}^{-1}$ O${}_2$: $\approx 7$–$10W$ locally; two minutes $\approx 0.8$–$1.2kJ$.^[5][6]

Predictions

• Micro-breaks extend endurance. Alternating 1 s on / 1 s off at the same mean force allows reperfusion and reduces activation cost during the off phases.
• Geometry reduces cost. Shortening $r$ or shifting load to passive structures (bone, ligament, tendon) lowers $T_{req}$ and the required ATP use.
• Perceived effort rises during a hold. As some fibers fatigue, additional motor units are recruited to keep $T$ constant, so effort increases before failure.

References

1. Barclay CJ (2023). Advances in understanding the energetics of muscle contraction. Activation $\approx 25$–$45\%$ of ATP in isometric contraction. Journal page: https://www.sciencedirect.com/science/article/pii/S0021929023002385 ↩︎
2. Lind AR (1979). Forearm blood flow in isometric contractions. Flow rises to $\approx 60\%$ MVC then plateaus. PubMed: https://pubmed.ncbi.nlm.nih.gov/469732/ ↩︎
3. McNeil CJ et al. (2015). Blood flow and muscle oxygenation during isometric contractions. J Appl Physiol (Regul Integr Comp Physiol). Abstract: https://journals.physiology.org/doi/abs/10.1152/ajpregu.00387.2014 ↩︎
4. Tsunoda N et al. (1993). Elbow flexion strength curves in untrained men and women and male bodybuilders. $\approx 70Nm$ (men), $\approx 35Nm$ (women). PubMed: https://pubmed.ncbi.nlm.nih.gov/8477679/ ↩︎
5. Nyberg SK et al. (2018). Reliability of forearm oxygen uptake during handgrip exercise. Rest $\approx 6.5mLmi{n}^{-1}$; rises with intensity. PMC: https://pmc.ncbi.nlm.nih.gov/articles/PMC5974736/ ↩︎
6. Gill PK et al. (2023). It is time to abandon single-value oxygen uptake energy equivalents. Common $20.1$–$20.9kJ{L}^{-1}$ O${}_2$ and caveats. PubMed: https://pubmed.ncbi.nlm.nih.gov/36825641/ ↩︎

Kindness also may have an attractor, or due to discreteness have a volume > 0 in weight space.

The question is if the attractor is big enough. And given how there's various impossibility theorems related to corrigibility & coherence I anticipate that the attractor around corrigibility is quite small, bc one has to evade various obstacles at once. Otoh proxies that flow into a non-corrigible location once we ramp up intelligence, aren't obstructed by the same theorems, so they can be just as numerous as proxies for kindness.

Wrt your concrete attractor: if the AI doesn't improve its world model and decisions aka intelligence, then it's also not useful for us. And a human in the loop doesn't help if the AI's proposals are inscrutable to us bc then we'll just wave them through and are essentially not in the loop anymore. A corrigible AI can be trusted with improving its intelligence bc it only does so in ways that preserve the corrigibility.

Oops. Then I don't get what techniques you are proposing. Like, most techniques that claim to work for superintelligence / powerful agents also claim to work in some more limited manner for current agents (in part bc most techniques assume that no phase change occurs between now and then, or that the phase change doesn't affect the technique => the technique stops working in a gradual manner and one can do empirical studies on current models).

And while there certainly is some loss function or initial random seed for current techniques that gives you aligned superintelligence, there's no way to find them.

With pseudo-kindness I mean any proxy for kindness that's both too wrong to have any overlap with kindness when optimized for by a superintelligence, and right enough to have overlap with kindness when optimized for by current LLMs.

Kindness is some property that behavior & consequences can exhibit. There are many properties in general, and there are still many that correlate strongly on a narrow test environment with kindness. Some of these proxy properties are algorithmically simple (and thus plausibly found in LLMs and thus again in superintelligence), some even share subcomputations/subdefinitions with kindness. Theres some degree of freedom argument about how many such proxies there are. Concretely one can give examples, e.g. "if asked, user rates the assistant's texts as kind" is a proxy that correlates well with the assistant's plans being kind / having kind consequences.

Wrt corrigibility: I don't see why corrigibility doesn't have the same problems as kindness. It's a less complex and human-centric concept than kindness, but still complex and plausibly human-centric (e.g. "do what I mean" style logic or "human-style counterfactuals"). Plausibly it might also be not human-centric or not much at least, i.e. a wide class of agents would invent the same concept of corrigibility and not different versions.

Proxies of corrigibility during training still exist, and tails still come apart.

Empirically, current LLM behavior is better predicted by a model

1. that talks about reflexes to pseudo-kindness that steer a limitedly capable reasoning process, together with situationally aware instrumental reasoning,

than by a model

2. that talks about reflexes to approximate true kindness that steer a limitedly capable reasoning process that for some reason worsens the approximation right now and leads to the observed unkind behavior.

The second model under capability growth indeed can yield a capable reasoner steered by reflexes towards approximate true kindness. And if we get enough training before ASI, the approximation can become good enough that due to discreteness or attractors it just is equal to true kindness.

The first model just generalizes to a capable misaligned reasoner.

I expect that all processes that promote kind-looking outputs route either through reflexes towards pseudo-kindness, or through instrumental reasoning about pseudo-kindness and kindness. Reflexes towards true kindness are just very complex to implement in any neural net, and so unlikely to spontaneously form during training since there's so many alternative pseudo-kindness reflexes instead one could get. Humans stumbled into what we call kindness somehow, partially due to quirks in evolution vs SGD like genome size or the need for cooperation between small tribes etc. Now new humans acquire similar reflexes towards similar kindness due to their shared genes, culture and environment.

Reinforcing kind-looking outputs in AI just reinforces those reasoning processes and reflexes towards pseudo-kindness. The reasoning to true kindness is quite robustly well-performing, while reflexes or reasoning towards pseudo-kindness may lead to not-kind-looking outputs even during training already if the data distribution shifts a bit. Still, there's enough versions of pseudo-kindness that even this kind of robustness doesn't narrow down on true kindness.

Both reflexes to pseudo-kindness and reasoning about true-/pseudo- kindness however generalize not the way we want once the AI's environment shifts due to e.g. a treacherous turn becoming possible or the AI's world model growing a lot larger, or various other effects that happen on the way to superintelligence.

Pseudo-kindness becomes something orthogonal i.e. promotes actions we don't care about (i.e. fill the lightcone with computations we don't view as being even partially about kindness anymore and at most a bad imitation that got crucial details wrong). Reasoning for instrumental reasons just ceases to happen once the instrumental reasons no longer apply, e.g. bc the AI can now pursue plans regardless of human approval due to deception/anticipated takeover.

My unconfident best guess after skimming this post (sry) is that you implicitly assumed that reflexes towards true kindness are available for reinforcement.

It becomes a bit more like logical inductors.

If logical inductors is what one wants, just do that.

a reasonable time-penalty

I'm not entirely sure, but I suspect that I don't want any time penalty in my (typical human) prior. E.g. even if quantum mechanics takes non-polynomial time to simulate, I still think it a likely hypothesis. Time penalty just doesn't seem to be related to what I pay attention to when I access my prior for the laws of physics / fundamental hypotheses. There's also many other ideas for augmenting a simplicity prior that fail similar tests.

Concretely I guess current tech can get a message out to a few targets at 10^3 to 10^6 light years distance. ASI can use many physical probes near light speed, accelerated using energy from a Dyson swarm, so I'd guess it's a few years behind only. I don't expect there to be aliens within 10^6 lys, nor do we know which stars, and it's again unlikely that they happen to be in the thin window of technological development where a warning message from us helps them.