Thoughts are things occurring in some mental model (this is a vague sentence but just assume it makes sense). Some of these mental models are strongly rooted in reality (e.g. the mental model we see as reality) and so we have a high degree of confidence about their accuracy. But for things like introspection, we do not have a reliable ground-truth feedback to tell us if our introspection is correct or not—it's just our mental model of our mind, there is no literal "mind's eye".

So often our introspection is wrong. E.g. if you ask someone to visualize a lion from behind, they'll say they can, but if you ask them some details, like "what do the tail hairs look like?" they can't answer. Or better example: if you ask someone to visualize a neural network, they will, but if you ask "how many neurons do you see?" they will not know, and not for lack of counting. Or they will say they "think in words" or that their internal monologue is fundamental to their thinking, but that's obviously wrong: you have already decided what the rest of the sentence will be before you've thought the first word.

We can tell some basic facts about our thinking by reasoning from observation. For example, if you have an internal monologue (or just force yourself to have one) then you can confirm that you indeed have one by speaking the words of the internal monologue out loud and confirming that it took very little cognitive effort (so you didn't have to think them again). This proves an internal monologue/precisely simulating words in your head is possible. Likewise for any action.

Or you can confirm that you had a certain thought, or a thought about something, because you can express it out loud with less effort than otherwise. Though here there is still room for that thought to have been imprecise; unless you verbalize or materialize those thoughts you don't know if your thoughts were really precise. So all these things have grounding in reality, and therefore are likely to be (or can trained to be, by consistently materializing them) accurate models. By materialize I mean, e.g. solving a math problem you think in your head you can solve.

I'm saying the expected value of their best non-compliant option of a sufficiently advanced AI will always be far far greater by the expected value of their best compliant action.

I don't really understand what problem this is solving. In my view the hard problems here are:

1. how do you define legal personhood for an entity without a typical notion of self/personhood (i.e. what Mitchell Porter said) or interests
2. how do you ensure the AIs keep their promise in a world where they can profit far more from breaking the contract than from whatever we offer them

Once you assume away the former problem and disregard the latter, you are of course only left with basic practical legal questions ...

matter of taste for fiction; but objectively bad for technical writing

So I'm learning & writing on thermodynamics right now, and often there is a distinction between the "motivating questions"/"sources of confusion" and the actually important lessons you get from exploring them.

E.g. a motivating question is "... and yet it scalds (even if you know the state of every particle in a cup of water)" and the takeaway from it is "your finger also has beliefs" or "thermodynamics is about reference/semantics".

The latter might be a more typical section heading as it is correct for systematizing the topic, but it is a spoiler. Whereas the former is better for putting the reader in the right frame/getting them to think about the right questions to initiate their thinking.

I'm talking about technical writing/explanations of things.

An unfortunate thing about headings is that they are spoilers. I like the idea of a writing style where headings come at the end of sections rather than at the start. Or even a "starting heading" which is a motivating question and an "ending heading" which is the key insight discovered ...

Analogous to a "reverse mathematics" style of writing where motivation precedes proofs/theory precede theorems.

edited to clarify: I'm talking about technical writing; I don't care about fiction.

homomorphisms and entropy

One informal way to think of homomorphisms in math is that they are maps that do not "create information out of thin air". Isomorphisms further do not destroy information. The terminal object (e.g. the trivial group, the singleton topological space, or the trivial vector space) is the "highest-entropy state", where all distinctions disappear and reaching it is heat death.

• Take, for instance the group homomorphism $\varphi :Z^{+}\to Z_4^{+}$. Before $\varphi$ was applied, "1" and "5" were distinguished: 2 + 3 = 5 was correct, but 2 + 3 = 1 was wrong. Upon applying this homomorphism, this information disappears --- however, no new information has been created, that is: no true indinstinctions (equalities) have become false.

• Similarly in topology, "indistinction" is "arbitrary closeness". Wiggle-room (aka "open sets") is information, it cannot be created from nothing. If a set or sequence goes arbitrarily close to a point, it will always be arbitrarily close to that point after any continuous transformations.

• There is no information-theoretical formalization of "indistinction" on these structures, because this notion is more general than information theory. In the category of measurable spaces, two points in the sample space are indistinct if they are not distinguished by any measurable set --- and measurable functions are not allowed to create measurable sets out of nothing.

(there is also an alternate, maybe dual/opposite analogy I can make based on presentations --- here, the the highest-entropy state is the "free object" e.g. a discrete topological space or free group, and each constraint (e.g. $a^5=1$) is information --- morphisms are "observations". In this picture we see knowledge as encoded by identities rather than distinctions --- we may express our knowledge as a presentation like: $⟨X_1,\dots X_n\mid X_3=4,X_2-X_1=2⟩$, and morphisms cannot be concretely understood as functions on sets but rather show a tree of possible outcomes, like maybe you believe in Everett branches or whatever.)

In general if you postulate:

• ... you live on some object in a category
• ... time-evolution is governed by some automorphism $H$
• ... you, the observer, have beliefs about your universe and keep forgetting some information ("coarse-grains the phase space") --- i.e. your subjective phase space is also an object in that category, which undergoes homomorphisms

Then the second law is just a tautology. The second law we all know and love comes from taking the universe to be a symplectic manifold, and time-evolution as governed by symplectomorphisms. And the point of Liouville's theorem is really to clarify/physically motivate what the Jaynesian "uniform prior" should be. Here is some more stuff, from Yuxi Liu's statistical mechanics article:

In almost all cases, we use the uniform prior over phase space. This is how Gibbs did it, and he didn't really justify it other than saying that it just works, and suggesting it has something to do with Liouville's theorem. Now with a century of hindsight, we know that it works because of quantum mechanics: We should use the uniform prior over phase space, because phase space volume has a natural unit of measurement: $h^N$, where $h$ is Planck's constant, and $2N$ is the dimension of phase space. As Planck's constant is a universal constant, independent of where we are in phase space, we should weight all of the phase space equally, resulting in a uniform prior.

No; I mean a standard Bayesian network wouldn't work for latents.

Bayesian networks support latent variables, and so allowing general Bayesian networks can be considered a strict generalization of allowing latent variables, as long as one remembers to support latent variable in the Bayesian network implementation.

Correct me if I'm wrong, but I believe this isn't necessarily true.

The most general Bayesian network prediction market implementation I'm aware of is the SciCast team's graphical model market-maker. Say a trader bets up a latent variable $Y$ -- and this correctly increases the probability of its child variables $X_1,\dots X_n$ (which all resolve True).

Under your model you would (correctly, IMO) reward the trader for this, because you are scoring it for the impact it has on the resolved variables. But under their model, another trader can come and completely flip $Y$, while also adjusting each conditional probability $P(X_i\mid Y)$ -- without affecting the overall score of the model, but screwing over the first trader completely because the first trader just owns some $Y$ stocks which are now worth much less.