The "duty to listen" could be qualified: We accept the duty to listen to value system V, if and only if V endorses the duty to keep listening. We cultivate an allergy to value systems that forbid further listening. The allergy enables us to consider (some) bad viral value systems, without getting trapped in them. 

I suspect there are sneakier forms of bad viral value systems that would trap us without triggering the allergy. I wonder what shape these have.

permanently adopt P

typo: permanently adopt V

Suppose I hand you a circuit C that I sampled uniformly at random from the set of all depth-n reversible circuits satisfying P. What is a reason to believe that there exists a short heuristic argument for the fact that this particular C satisfies P?

mathematicians who worked on FrontierMath would possibly have not contributed to this if they knew about the funding and exclusive access.

I'm a mathematician who contributed to FrontierMath. 

Speaking just for myself: From the beginning it was clear reading between the lines that the project had an industry sponsor, with OAI being an obvious guess. I judged the project as having a less favorable safety/capabilities tradeoff than my other research, to the point where I drafted, but did not ultimately send, an email bowing out. In hindsight, I think staying in was the right call: It was a good learning experience for me to work on something directly impactful, even if the sign of the impact is ambiguous.

My main updates are: 

• Benchmarks are super impactful! My main research is on dispositional AI safety, so now I'm thinking about dispositional benchmarks.
• Proximity to power is confusing! It’s good to start wrestling with that confusion now, while the stakes are still low.

So, boundaries enable cooperation, by protecting BATNA.

Would you say there is a boundary between cell and mitochondria?

In the limit of perfect cooperation, the BATNA becomes minus infinity and the boundary dissolves.

Thanks all for responding! The meetup will be this Thursday, any other Ithaca locals DM for details!

This is an obvious point, but: Any goal is likely to include some variance minimization as a subgoal, if only because of the possibility that another entity (rival AI, nation state, company) with different goals could take over the world.  If an AI has the means to take over the world, then it probably takes seriously the scenario that a rival takes over the world. Could it prevent that scenario without taking over itself?

money is probably much less valuable after AGI than before, indeed practically worthless.

I think this overstates the case against money. Humans will always value services provided by other humans, and these will still be scarce after AGI. Services provided by humans will grow in value (as measured by utility to humans) if AGI makes everything else cheap.  It seems plausible that money (in some form) will still be the human-to-human medium of exchange, so it will still have value after AGI.

If Alice and Bob are talking to each other as they deliberate

I think this is a typo, it should say "compete" instead of "deliberate".

I worry about persuasion becoming so powerful that it blocks deliberation: How can Alice know whether Bob (or his delegated AI) is deliberating in good faith or trying to manipulate her?

In this scenario, small high-trust communities can still deliberate, but mutual mistrust prevents them from communicating their insights to the rest of the world.

I think this is possible and it doesn’t require AI. It only requires a certain kind of "infectious Turing machine" described below. 

Following Gwern’s comment, let’s consider first the easier problem of writing a program on a small portion of a Turing machine’s tape, which draws a large smiley face on the rest of the tape. This is easy even with the *worst case* initialization of the rest of the tape.  Whereas our problem is not solvable in worst case, as pointed out by Richard_Kennaway. 

What makes our problem harder is errors caused by the random environment. We could model these errors by spontaneous random changes in the letters written on the Turing machine's tape.  But it’s easy to make a computation robust to such errors:  A naive way to do it is to repeat each square 100 times and assign a repair bot to repeatedly scan the 100 squares, compute majority, and convert the minority squares back to the majority letter. This is not so different from how your laptop repeatedly scans its memory to prevent errors.

Now we get to what I think is the hard part of the problem: How to organize a random environment into squares of Turing machine tape? I don’t know how to do it in Conway's Life, but I would guess it's possible. An interesting question is whether *any* cellular automaton that supports a Turing machine can also support an infectious Turing machine. Again I would guess yes.

Does an infectious Turning machine require AI? There is certainly something lifelike about a device that gradually converts its environment into squares of its own tape. The tape squares reproduce, but they hardly need to be intelligent.

But okay, we’re setting the bar low by competing against a *random* environment. As Oscar_Cunningham points out, a random environment is unlikely to produce anything smart. If we set the bar higher by asking the infection to compete against an environment containing life or intelligence, then it might need to be significantly smarter in order to take over the rest.