There's this intro series by @Alex Lawsen.

That seems an odd motte-and-bailey style explanation (and likely, belief. As you say, misgeneralized).

From my side or theirs?

Huh. Intuitively this doesn't feel like it rises to the quality needed for a post, but I'll consider it. (It's in the rats tail of all the thoughts I have about subagents :-))

(Also: Did you accidentally a word?)

Only to a small extent.

What then prevents humans from being more terrible to each other? Presumably, if the vast majority of people are like this, and they know that the vast majority of others are also like this, up to common knowledge, I don't see how you'd get a stable society in which people aren't usually screwing each other a giant amount.

Prompted by this post, I think that now is a very good time to check how easy it is for someone (with access to generative AI) impersonating you to get access to your bank account.

On a twitter lent at the moment, but I remember this thread. There's also a short section in an interview with David Deutsch:

So all hardware limitations on us boil down to speed and memory capacity. And both of those can be augmented to the level of any other entity that is in the universe. Because if somebody builds a computer that can think faster than the brain, then we can use that very computer or that very technology to make our thinking go just as fast as that. So that's the hardware.
[…]
So if we take the hardware, we know that our brains are Turing-complete bits of hardware, and therefore can exhibit the functionality of running any computable program and function.

and:

So the more memory and time you give it, the more closely it could simulate the whole universe. But it couldn't ever simulate the whole universe or anything near the whole universe because it is hard for it to simulate itself. Also, the sheer size of the universe is large.

I think this happens when people encounter the Deutsch's claim that humans are universal explainers, and then misgeneralize the claim to Turing machines.

So the more interesting question is: Is there a computational class somewhere between FSAs and PDAs that is able to, given enough "resources", execute arbitrary programs? What physical systems do these correspond to?

Related: Are there cognitive realms? (Tsvi Benson-Tilsen, 2022)

Yes, I was interested in the first statement, and not thinking about the second statement.

Not "humans are a general turing-complete processing system", that's clearly false

Critical rationalists often argue that this (or something very related) is true. I was not talking about whether humans are fully implementable on a Turing machine, that seems true to me, but was not the question I was interested in.

Could you explain more about being coercive towards subagents? I'm not sure I'm picking up exactly what you mean.

A (probably-fake-)framework I'm using is to imagine my mind being made up of subagents with cached heuristics about which actions are good and which aren't. They function in a sort-of-vetocracy—if any one subagent doesn't want to engage in an action, I don't do it. This can be overridden, but doing so carries the cost of the subagent "losing trust" in the rest of the system and next time putting up even more resistance (this is part of how ugh fields develop).

The "right" way to solve this is to find some representation of the problem-space in which the subagent can see how its concerns are adressed or not relevant to the situation at hand. But sometimes there's not enough time or mental energy to do this, so the best available solution is to override the concern.

This seems right. One thing I would say is that kind of surprisingly it hasn't been the most aversive tasks where the app has made the biggest difference, it's the larger number of moderately aversive tasks. It makes expensive commitments cheap and cheap commitments even cheaper, and for me it has turned out that cheap commitments have made up most of the value.

Maybe for me the transaction costs are still a bit too high to be using commitment mechanisms, which means I should take a look at making this smoother.