Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This is a special post for short-form writing by davidad. Only they can create top-level comments. Comments here also appear on the Shortform Page and All Posts page.

I want to go a bit deep here on "maximum entropy" and misunderstandings thereof by the straw-man Humbali character, mostly to clarify things for myself, but also in the hopes that others might find it useful. I make no claim to novelty here—I think all this ground was covered by Jaynes (1968)—but I do have a sense that this perspective (and the measure-theoretic intuition behind it) is not pervasive around here, the way Bayesian updating is.

First, I want to point out that entropy of a probability measure p is only definable relative to a base measure μ, as follows:

Hμ(p)=−∫Xdpdμ(x)logdpdμ(x)dμ(x)(The derivatives notated here denote Radon-Nikodym derivatives; the integral is Lebesgue.) Shannon's formulae, the discrete H(p)=−∑ip(xi)logp(xi) and the continuous H(p)=−∫Xp(x)logp(x)dx, are the special cases of this where μ is assumed to be counting measure or Lebesgue measure, respectively. These formulae actually treat p as having a subtly different

typethan "probability measure": namely, they treat it as a density with respect to counting measure (a "probability mass function") or a density with respect to Lebesgue measure (a "probability density function"), and implicitly supply the corresponding μ.If you're familiar with Kullback–Leibler divergence (DKL), and especially if you've heard DKL called "relative entropy," you may have already surmised that Hμ(p)=−DKL(p||μ). Usually, KL divergence is defined with both arguments being

probabilitymeasures (measures that add up to 1), but that's not required for it to be well-defined (what is required is absolute continuity, which is sort of orthogonal). The principle of "maximum entropy," or argmaxpHμ(p), is equivalent to argminpDKL(p||μ). In the absence of additional constraints on p, the solution of this is p=μ, so maximum entropy makes sense as a rule for minimum confidenceto exactly the same extent that the implicit base measureμmakes sense as a prior. The principle of maximum entropy should really be called "the principle of minimum updating", i.e., making a minimum-KL-divergence move from your prior μ to your posterior p when the posterior is constrained to exactly agree with observed facts. (Standard Bayesian updating can be derived as a special case of this.)Sometimes, the structure of a situation has some symmetry group with respect to which the situation of uncertainty seems to be invariant, with classic examples being relabeling heads/tails on a coin, or arbitrarily permuting a shuffled deck of cards. In these examples, the requirement that a prior be invariant with respect to those symmetries (in Jaynes' terms, the principle of transformation groups) uniquely characterizes counting measure as the only consistent prior (the classical principle of indifference, which still lies at the core of grade-school probability theory). In other cases, like a continuous roulette wheel, other Haar measures (which generalize both counting and Lebesgue measure) are justified. But taking "indifference" or "insufficient reason" to justify using an invariant measure as a prior

in an arbitrary situation(as Laplace apparently did) is fraught with difficulties:improper prior: it is a non-probability measure because its integral (formally, ∫∞−∞1dμ(x)) is infinite. If we're talking about forecasting the timing of a future event, (0,∞) is a very natural space, but ∫∞01dμ(x) is no less infinite. Discretizing into year-sized buckets doesn't help, since counting measure on N is also infinite (formally, ∑ωi=01=∞). In the context of maximum entropy, using an infinite measure for μ means that thereisno maximum-entropy p—you can always get more entropy by spreading the probability masseven thinner.andalso impose a nothing-up-my-sleeve outrageous-but-finite upper bound, like the maximum`binary64`

number at around 1.8×10308? Counting measure on {i:N|i<1.8×10308} can be normalized into a probability measure, so what stopsthatfrom being a reasonable "unconfident" prior? Sometimes this trick can work, but the deeper issue is thatthe original symmetry-invariance argument that successfully justifies counting measure for shuffled cards just makes no sense here. If one relabels all the years, say reversing their order, the situation of uncertainty is decidedlynotequivalent.I think Humbali's confusion can be partially explained as conflating an invariant measure and a prior—in

bothdirections:implicitly uses a translation-invariant base measure as a priorwhen he claims as absolute a notion of "entropy" which is actually relative to that particular base measure. Something like this mistake was made by both Laplace and Shannon, so Humbali is in good company here—but already confused, because translation on the time axis is not a symmetry with respect to which forecasts ought to be invariant.implicitly uses his (socially-driven) prior as a base measure, when he says "somebody with a wide probability distribution over AGI arrival spread over the next century, with a median in 30 years, is in realistic terms about as uncertain as anybody could possibly be." Assuming he's still using "entropy" at all as the barometer of virtuous unconfidence, he's now saying that the way to fix the absurd conclusions of maximum-entropy relative to Lebesgue measure is that onereallyought to measure unconfidence with respect to a socially-adjusted "base rate" measure, which just happens to be his own prior. (I think the lexical overlap between "base rate" and "base measure" is not a coincidence.) This second position is more in bad-faith than the first because it still has the bluster of objectivity without any grounding at all, but it has more hope of formal coherence: one can imagine a system of collectively navigating uncertainty where publicly maintaining one's own epistemic negentropy,explicitlyrelative to some kind of social median, comes at a cost (e.g. hypothetically or literally wagering with others).There is a bit of motte-and-bailey uncovered by the bad-faith in position 2. Humbali all along primarily wants to defend his prior as unquestionably reasonable (the bailey), and when he brings up "maximum entropy" in the first place, he's retreating to the motte of Lebesgue measure, which seems to have a formidable air of mathematical objectivity about it. Indeed, by its lights, Humbali's own prior does happen to have more entropy than Eliezer's, though Lebesgue measure fails to support the full bailey of Humbali's actual prior. However, in this case even the motte is not defensible, since Lebesgue measure is an improper prior and the translation-invariance that might justify it simply has no relevance in this context.

Meta: any feedback about how best to make use of the channels here (commenting, shortform, posting, perhaps others I'm not aware of) is very welcome; I'm new to actually contributing content on AF.

Ha, I was just about to write this post. To add something, I think you can justify the uniform measure on bounded intervals of reals (for illustration purposes, say [0,1]) by the following argument: "Measuring a real number x∈[0,1]" is obviously simply impossible if interpreted literally, containing an infinite amount of data. Instead this is supposed to be some sort of idealization of a situation where you can observe "as many bits as you want" of the binary expansion of the number (choosing another base gives the same measure). If you now apply the principle of indifference to each measured bit, you're left with Lebesgue measure.

It's not clear that there's a "right" way to apply this type of thinking to produce "the correct" prior on N (or R or any other non-compact space.

Given any particular admissible representation of a topological space, I do agree you can generate a Borel probability measure by pushing forward the Haar measure of the digit-string space ΣN (considered as a countable product of ω copies of Σ, considered as a group with the modular-arithmetic structure of Z/|Σ|) along the representation. This construction is studied in detail in (Mislove, 2015).

But, actually, the representation itself (in this case, the Cantor map) smuggles in Lebesgue measure, because each digit happens to cut the target space "in half" according to Lebesgue measure. If I postcompose, say, x↦√x after the Cantor map, that is also an admissible representation of [0,1], but it no longer induces Lebesgue measure. This works for any continuous bijection, so

anyabsolutely continuous probability measure on [0,1] can be induced by such a representation. In fact, this is why the inverse-CDF algorithm for drawing samples from arbitrary distributions, given only uniform random bits, works.That being said, you

canapply this to non-compact spaces. I could get a probability measure on R via a decimal representation, where, say, the number of leading zeros encodes the exponent in unary and the rest is the mantissa. [Edit: I didn't think this through all the way, and it can only represent real numbers ≥1. No big obstacle; post-compose x↦log(x−1).] The reason there doesn't seem to be a "correct" way to do so is that, because there's no Haar probability measure on non-compact spaces (at least, usually?), there's no digit representation that happens to cut up the space "evenly" according to such a canonical probability measure.Right, that's essentially what I mean. You're of course right that this doesn't let you get around the existence of nonmeasurepreserving automorphisms. I guess what I'm saying is that, if you're trying to find a prior on [0,1], you should try to think about what system of finite measurements this is idealizing, and see if you can apply a symmetry argument to those bits. Which isn't always the case! You can only apply the principle of indifference if you're actually indifferent. But

ifit's the case that X∈[0,1] is generated in a way where "there's no reason to suspect that any bit should be 0 rather than 1, or that there should be correlations between the bits", then it's of course not the case that √X has this same property. But of course you still need to look at the actual situation to see what symmetry exists or doesn't exist.Haar measures are a high-powered way of doing this, I was just thinking about taking the iterated product measure of the uniform probability measure on {0,1} (justified by symmetry considerations). You can of course find maps from {0,1}∞ to all sorts of spaces but it seems harder to transport symmetry considerations along these maps.

Out of curiosity, this morning I did a literature search about "hard-coded optimization" in the gradient-based learning space—that is, people deliberately setting up "inner" optimizers in their neural networks because it seems like a good way to solve tasks. (To clarify, I don't mean deliberately trying to make a general-purpose architecture learn an optimization algorithm, but rather, baking an optimization algorithm into an architecture and letting the architecture learn what to do with it.)

Why is this interesting?doeshelp competitiveness, we should expect to see some of the relevant competitors trying to do it on purpose.helpsafety to bake in inference-time optimization on purpose, since we can better control and understand optimization when it's engineered—assuming that engineering it doesn't sacrifice task performance (so that the incentive for the base optimizer to evolve ade novomesa-optimizer is removed).weaklydifferential tech development in the sense that it accelerates safe AImorethan it accelerates unsafe AI (although it accelerates both). I'm not confident enough about this to say that it's a good direction to work on, but I am saying it seems like a good direction to be aware of and occasionally glance at.Anyway, here's (some of)

what I found:The Perils of Learning Before Optimizing.”AAAI 2022, December 16, 2021wantto incorporate an optimizer into a learning system, rather than learning a predictive model and then applying an engineered optimizer.Differentiable Convex Optimization Layers.”NIPS 2019, October 28, 2019Efficient Differentiable Quadratic Programming Layers: An ADMM Approach.”ArXiv:2112.07464, December 14, 2021Differentiable Dynamic Programmingfor Structured Prediction and Attention.”ICML 2018, February 20, 2018Deep Equilibrium Architectures for Inverse Problemsin Imaging.”ArXiv:2102.07944, June 2, 2021∂-Explainer: Abductive Natural Language Inference via Differentiable Convex Optimization."ArXiv:2105.03417, May 17, 2021Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models.”NIPS 2021, November 12, 2021physics predictor, specifically for contact dynamics—based on the physical principle that contact forces maximize the rate of energy dissipation. The physics predictor is then wrapped in a gradient-based optimization to compute optimal actions starting from a given initial state.Learning Convex Optimization Control Policies.”ArXiv:1912.09529, December 19, 2019Fast Adaptation of Manipulator Trajectories to Task PerturbationBy Differentiating through the Optimal Solution.”ArXiv:2011.00488, November 1, 2020Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, 316–25. BuildSys ’19. New York, NY, USA: Association for Computing Machinery, 2019Here are some other potentially relevant papers I haven't processed yet:

ArXiv:2201.01347 [Cs, Eess], January 7, 2022. http://arxiv.org/abs/2201.01347.ArXiv:2102.05791 [Cs], September 9, 2021. http://arxiv.org/abs/2102.05791.Proceedings of the 35th International Conference on Machine Learning, 4732–41. PMLR, 2018. https://proceedings.mlr.press/v80/srinivas18b.html.Among computational constraints, I think the most significant/fundamental are, in order,

For various collective-epistemics and cooperative-decision-making endeavours, I think a key technical enabler might be DVCS for structured data. To that end, I am interested in funding work in this direction. Aside from being in a position to allocate some funding, I think I have some comparative advantage in a broad inside-view awareness of potentially relevant theoretical footholds, and this post is intended to start unfurling that inside view, initially as a list of links. People who I fund to work on this should read the abstracts of all of these papers, pay special attention to those marked with (!), skim/read further as they see fit, and cite them in their writeups (at least under "related work"). I'm posting this here as part of a general policy of using this platform for any of my even-vaguely-technical output that goes beyond tweets.

hypernets, a combinatorial representation of string diagrams, defined alongside a double-pushout-rewriting scheme for them in section 4 of Functorial String Diagrams for Reverse-mode Automatic Differentiation. Ideally, this should be a special case of the above.anydecentralized system.“Concern, Respect, and Cooperation” is a contemporary moral-philosophy book by Garrett Cullity which advocates for a pluralistic foundation of morality, based on three distinct principles:

What I recently noticed here and want to write down is a loose correspondence between these different foundations for morality and some approaches to safe superintelligence:

Cullity argues that none of his principles is individually a satisfying foundation for morality, but that all four together (elaborated in certain ways with many caveats) seem adequate (and maybe just the first three). I have a similar intuition about AI safety approaches. I can’t yet make the analogy precise, but I feel worried when I imagine corrigibility alone, CEV alone, bargaining alone (whether causal or acausal), or Earth-as-wildlife-preserve; whereas I feel pretty good imagining a superintelligence that somehow balances all four. I can imagine that one of them might suffice as a foundation for the others, but I think this would be path-dependent at best. I would be excited about work that tries to do for Cullity’s entire framework what CEV does for pure single-agent utilitarianism (namely, make it more coherent and robust and closer to something that could be formally specified).

Useful primitives for incentivizing alignment-relevant metrics without compromising on task performance might include methods like Orthogonal Gradient Descent or Averaged Gradient Episodic Memory, evaluated and published in the setting of continual learning or multi-task learning. Something like “answer questions honestly” could mathematically be thought of as an additional task to learn, rather than as an inductive bias or regularization to incorporate. And I think these two training modifications are quite natural (I just came to essentially the same ideas independently and then thought “if either of these would work then surely the multi-task learning folks would be doing them?” and then I checked and indeed they are). Just some more nifty widgets to add to my/our toolbox.

Re: alignment tasks in multi-task settings. I think this makes a lot of sense. Especially in worlds where we have a lot of ML/AI systems doing a bunch of different things, even if they have very different specific tasks, the "library" of alignment objectives is probably widely shared.

What happens if you ask a logical inductor whether ππππ is an integer? What's your own subjective probability?

This is one question where my odds are pretty far from being coherently Bayesian; I would bet against only up to around 100:1, but I wouldn't bet in favor until somewhere around 8e7:1.

Why is that not Bayesian? The decision to bet is going to include a term for your counterparty’s willingness to bet, which is some evidence.

One way to overcome this in thought experiments is to frame it as a line with no spread, and no neutral option. At, say, 150k:1, which side would you bet? Then adjust to where you'd SET a line where you had to accept a wager on either side.