I wouldn't get too hung up on the word 'regulator'. It's used in a very loose way here, as in common in old cybernetics-flavoured papers.

Human slop (I'm referring to those old cybernetics papers rather than the present discussion) has no more to recommend it than AI slop. "Humans Who Are Not Concentrating Are Not General Intelligences", and that applies not just to how they read but also how they write.

If you are thinking of something like 'R must learn a strategy by trying out actions and observing their effect on Z' then this is beyond the scope of this post! The Good Regulator Theorem(s) concern optimal behaviour, not how that behaviour is learned.

What I am thinking of (as always when this subject comes up) is control systems. A room thermostat actually regulates, not merely "regulates", the temperature of a room, at whatever value the user has set, without modelling or learning anything. It, and all of control theory (including control systems that do model or adapt), fall outside the scope of the supposed Good Regulator Theorem. Hence my asking for a practical example of something that it does apply to.

Puzzle 7a. I have two children, of whom at least one is a boy. Their names are Alex and Sam. (In this fictional puzzle, these names communicate no information about gender.) Alex is a boy. What is the probability that Sam is a boy?

Puzzle 1a. I have two children, of whom at least one is a boy. Now I shall toss a coin and accordingly choose one of the two children. (Does so.) The child I chose is a boy. What is the probability that the other is a boy?

I have difficulty coming up with a practical example of a system such as described in the section "The Original Good Regulator Theorem". Can you provide some examples?

Here are some of the problems I have in understanding the setup.

1. Minimising the entropy of Z says that Z is to have a narrow distribution, but says nothing about where the mean of that distribution should be. This does not look like anything that would be called "regulation".

2. Time is absent from the system as described. Surely a "regulator" should keep the value of Z near constant over time?

3. The value of Z is assumed to be a deterministic function of S and R. Systems that operate over time are typically described by differential equations, and any instant state of S and R might coexist with any state of Z.

4. R has no way of knowing the value of Z. It is working in the dark. Why is it hobbled in this way?

5. Z is assumed to have no effect on S.

6. There are no unmodelled influences on Z. In practice, there are always unmodelled influences.

These problems still apply when the variables X and N are introduced.

One has to consider the data-generating process behind puzzles such as these. I am going to assume throughout that it is what seems to me the simplest consistent with the statement of the problems.

1. Every child is independently equally likely to be a boy or a girl.

2. Every birth day of the week is equally and independently likely.

3. The puzzle narrator is chosen equally at random from all members of the population for which the statements that the narrator makes are true.

One can imagine all manner of other data-generating processes, which in general will give different answers. Most quibbles with such problems come down to imagining different processes, especially for statement 3. Some examples are in the OP and the comments. However, the above assumptions seem the simplest, and the ones intended by people posing such problems. If this question came up on an exam, I would be sure to begin my answer with the above preamble.

Puzzles 1 and 2 can be unified by considering a common generalisation to there being N days in a week, one of which is called Tuesday. Puzzle 1 has N=1. Puzzle 2 has N=7.

When two children are born, there are 4 N^2 possibilities for their sexes and birthdays. Which of these are left after being given the information in the puzzle?

The sexes must be either boy-boy, boy-girl, or girl-boy.

For boy-boy, there are 2N-1 ways that at least one is born on a Tuesday. For boy-girl, there are N ways the boy could be born on a Tuesday. For girl-boy, by symmetry also N.

So the probability of boy-boy is (2N-1)/(2N-1 + N + N) = (2N-1)/(4N-1).

For N=1 this is 1/3, which is the generally accepted answer to Puzzle 1.

For N=7, it is 13/27. In general, the larger N is, the closer the probability of the other being a boy is to 1/2.

For Puzzle 3, the obvious answer is that before learning the day, it's as Puzzle 1, 1/3, and after, it's as Puzzle 2, 13/27. Is obvious answer correct answer? I have (before reading the OP's solution) not found a reason not to think so. The OP says, by a different intuitive argument, that the obvious answer is 1/3, and that obvious answer is correct answer, but WilliamKiely's comment raises a doubt, finding ambiguity in the statement of the data-generating process. This leaves me as yet undecided.

I started thinking about the ways that extra information (beyond "I have two children, at least one of whom is a boy") affects the probability of two boys, and came up with these:

Puzzle 4. I have two children, of whom at least one is a boy with black hair. In this fictional puzzle, everyone knows that all the children in a family have the same hair colour. What is the probability that the other is also a boy?

Puzzle 5. I have two children, of whom at least one is a boy born on a Tuesday. In this fictional puzzle, everyone knows that two consecutive children never have the same birth day. What is the probability that the other is also a boy?

Puzzle 6. I have two children. The elder one is a boy. What is the probability that the younger is also a boy?

Puzzle 7. I have two children, called Alex and Sam. (In this fictional puzzle, these names communicate no information about gender.) Alex is a boy. What is the probability that Sam is a boy?

Puzzle 8. I have two children, of whom at least one is a boy. What is the probability that the elder child is a boy?

I turned the knob up to 11 on demandingness in the Insanity Wolf Sanity Test. (The first section, "Altruism" is the most relevant here.)

Of course, I intend that section as a critique of the monstrous egregore that (in my opinion) is utilitarianism. But that is the true denial of supererogation. If you don't want to go as far as Insanity Wolf, where do you stop and why? Or do you go modus ponens to my modus tollens and accept the whole thing?

I think it was strategically valuable for the early growth of EA that leaders denied its demandingness, but I worry some EAs got unduly inoculated against the idea.

You mean, they lied, then people believed the lies? Or are the lies for the outer circle and the public, while the inner circle holds to the secret, true doctrine of all-demandingness? I am not playing Insanity Wolf with that suggestion. Peter Singer himself has argued that the true ethics must be kept esoteric.

If you can give the AGI any terminal goal you like, irrespective of how smart it is, that’s orthogonality right there.

I don’t think the problem is well posed. It will do whatever most effectively goes towards its terminal goal (supposing it to have one). Give it one goal and it will ignore making paperclips until 2025; give it another and it may prepare in advance to get the paperclip factory ready to go full on in 2025.

When driving a car, I navigate using Satellite View!

It took me a moment to realise you were talking about Google Maps on a phone. Best of luck trying that in my part of the world. Default view, Satellite view, without a phone signal you don’t have either, and I usually don’t in the countryside..

Just an additional observation here. Similar concerns can also arise without AI being involved.

There's an artist whose works I've seen, and I am wondering if he is cheating. (Not a public figure that anyone here is likely to have heard of.) He makes strikingly photorealistic paintings on canvas. I have seen these canvases close enough to be sure that they really are painted, not printed. And I know that there are artists who indeed perform artistic miracles of photorealism. But I still have the suspicion that this artist is doing a paint-over of something printed onto the canvas. Perhaps a photograph of his intended scene, reduced to grayscale, edge-enhanced, and printed very faintly, to resemble and serve the same purpose as an artist's preliminary pencil underdrawing. Then he might paint using a full colour print of the image alongside as reference.

This suspicion somewhat devalues his art in my eyes. This method (if that is what he is doing) eliminates the need for any skill at draughtsmanship, but at the same time it removes the possibility of any creativity in the draughtsmanship. And if he is then slavishly copying by hand the original photograph onto this automated underdrawing, then he is just being a human photocopier. The only human thing left is the technical skills of mixing and blending colours, and handling a brush.

The verbose writing style makes me wonder if an LLM was used in the writing process?

Definitely. See also, which ironically has a similar verbosity.