I think the overall point you're making is intriguing, and I could see how it might alter my home behaviour if I considered it more deeply. But I also strongly disagree with the following:

Just about every work behavior is an example of bad home behavior

There is a bunch of "work behavior" that has been very useful – in the right measure – for my personal life:

• Task Management – This cut down on the time I spend on "life admin". 
• Scheduling – Reaching out with "let's find an evening to play tennis" helps me increase the number of fulfilling activities I do with friends.
• Prioritization – Day-to-day life can obscure what's really important. Thinking about what I really value and want to achieve can make my life more meaningful.
• Creating Spreadsheets & Documents – Apart from the obvious use case in personal finance, Spreadsheets are also very valuable to me for evaluating crucial life decisions ("Where should we move?"). I use documents for private events I'm organising (e.g. a weekend trip to the mountains with friends).

Maybe some of them are too obvious and common. But they are things that my grandmother wouldn't have done – and I suspect that they are mostly derived from work culture.

How useful do people think it would be to have human-in-the-loop (HITL) AI systems? 

What's the likeliest failure mode of HITL AI Systems? And what kind of policies are mostly likely to help against those failure modes?

If we assume that HITL AI Systems are not going to be safer once we reach unaligned AGI. Could it maybe give use tools/information/time-steps to increase the chance that AGI will be aligned?

I was thinking of money. :)

Interesting. Can you give us a sense of how much those asks (offer to pay for the extra labour) end up costing you?

Also: The last name is "Von Almen" not "Almen"

I always assumed that "Why don't we give Terence Tao a million dollars to work on AGI alignment?" was using Tao to refer to a class of people. Your comment implies that it would be especially valuable for Tao specifically to work on it. 

Why should we believe that Tao would be especially likely to be able to make progress on AGI alignment (e.g. compared to other recent fields medal winners like Peter Scholze)?

Just saw the inverse question was already asked and answered.

[I think this is more anthropomorphizing ramble than concise arguments. Feel free to ignore :) ]

I get the impression that in this example the AGI would not actually be satisficing. It is no longer maximizing a goal but still optimizing for this rule. 

For a satisficing AGI, I'd imagine something vague like "Get many paperclips" resulting in the AGI trying to get paperclips but at some point (an inflection point of diminishing marginal returns? some point where it becomes very uncertain about what the next action should be?) doing something else. 

Or for rules like "get 100 paperclips, not more" the AGI might only directionally or opportunistically adhere. Within the rule, this might look like "I wanted to get 100 paperclips, but 98 paperclips are still better than 90, let's move on" or "Oops, I accidentally got 101 paperclips. Too bad, let's move on".

In your example of the AGI taking lots of precautions, the satisficing AGI would not do this because it could be spending its time doing something else.

I suspect there are major flaws with it, but an intuition I have goes something like this:

• Humans have in some sense similar decision-making capabilities to early AGI.
• The world is incredibly complex and humans are nowhere near understanding and predicting most of it. Early AGI will likely have similar limitations.
• Humans are mostly not optimizing their actions, mainly because of limited resources, multiple goals, and because of a ton of uncertainty about the future. 
• So early AGI might also end up not-optimizing its actions most of the time.
• Suppose we assume that the complexity of the world will continue to be sufficiently big such that the AGI will continue to fail to completely understand and predict the world. In that case, the advanced AGI will continue to not-optimize to some extent.
  • But it might look like near-complete optimization to us.