I think goal factoring may in turn have been inspired by Geoff Anders’ goal mapping

I think this was mostly convergent development, rather than a clear lineage.

Though Geoff did teach a version of Goal Factoring that was much more like CT-charting at some early CFAR workshops.

This seems like the opposite of a disagreement to me? Am I missing something?

 

Uh, I think I don't want to leave a list of people, who didn't opt in to being a topic of discussion. But Eliezer has already been mentioned, as an example. We could talk privately about other specific cases.

My guess is that the people who are unusually disembodied that you're thinking of probably suppress a kind of contempt and/or anger at other people who don't have so much will-to-Goodness.

I think...maybe yes, of all the men that I'm thinking of, but no of all the women that I'm thinking of? Modulo, it doesn't seem very suppressed. 

Given that humans are our only existing example of decent agents, I think one obvious sanity check for proposed measures of AI agency is whether they are helpful for characterizing variation in human agency.

This seems like an obvious and apt question to ask, but I don't think it's an obvious sanity check, in the sense that "if a measure doesn't pass this check, that's a strong sign that it's not capturing what we care about."

AIs are different than human minds! I think it's not at all surprising if they have different limiting constraints and therefore very different "capability profiles."

Like, for humans, working memory is an important constraint on many of the complicated intellectual operations that we do. And working memory correlates with overall cognitive ability.

When you try to measure human intelligence , and figure out what it is made of, working memory is something like one of the major factors that falls out of the factor analysis. 

But if we imagine aliens that have vastly larger working memories (or "context windows") than humans. These aliens might still vary in working memory capacity, but it might be close to irrelevent for predicting their overall cognitive performance, because the bottlenecks on their cognitive ability are something else entirely.

I think that's exactly the situation we're in with the AIs. Their minds are of a quite different shape than ours, and so good proxy metrics for human capability won't generalize to AIs or vis versa. 



Overall, great post.
 

I claim that there are fairly solid arguments that address your three concerns. Do you feel satisfied by the answers already given, in the comments, here? Or should I reply to them at length?

Alternatively, I'd be up for talking through it, synchronously, over a video call (and posting the recording?) if that seems better for you.

After encountering a number of posts wondering how outsiders were responding to the book, I thought it might be valuable for me to write mine down.

Thank you!

I loved reading "My loose priors going in" and "To skip ahead to my posteriors". Great, concise, way to capture the impact of the book for you. More reviews should try that format.

[I'm not sure how to productively engage here, because it seems like it's hard to do more than throw our differing impressions at each other, and we should expect to have wildly differing experiences depending on the details of our own psychologies, which are both putting a lot of selection on what social and psychological observations we make of others, and also how we interpret those observations. It's a bit like discussing shapes in clouds, except with more of an annoying insistence that the shapes we're seeing correspond to something real in the territory and not just our projections.] 

People who have a very strong "will-to-Goodness" don't necessarily have very strong/extreme shadows, but often do, because they created the very strong will-to-Goodness by strongly suppressing their antisocial desires, which then strongly polarized those desires.

My impression is that this is mostly wrong, or at least least wrong with regard to what I meant by "will-to-Goodness", though I agree that there are some dynamics like this in play around these topics.

A lot of particularly scrupulous people's moral behavior routes through social or parental acceptance (who behave in some approved pattern of behavior out of an insecurity). I think that often does involve suppressing their shadow, maybe even in the typical case.

I've never known a person with vibrant will-to-Goodness (as I mean it, and as I am able to detect it) who was, to my knowledge, motivated that way.^[1]

Further, those people have various issues (from poor emotional regulation, to social-epistemic confidence-anxiety, to various blindspots), but they definitely don't read to me to be repressing socially-disapproved shadow parts more than most people. Almost the opposite.

However, almost 100% of the cases I'm thinking of are people who are unusually disembodied, which is at least suggestive of repressing something, but not suppression of antisocial desires. That might be a kind of shadow, but it's not a central example of what comes to mind when people talk about shadow work, and it's not the kind of thing that is sated by BDSM.

 

1. ^{^}

   I'm on weaker ground here, but I speculate that the thing that I'm detecting is a morality that is grounded in their own desires for the world, or for how to be, instead of a morality that's motivationally grounded in a social acceptance desire.

I wonder if I'll learn that people can be deeply good while having violent sexual preferences at the same time.

I resonate with this as a question.

I have an outstanding confusion about this. Specifically, the Sequences ring with vibrant and determined will-to-Goodness, but also Eliezer apparently likes hurting people (or something like that?) in bed. To me, these would seem, if not incompatible, than at least in tension. 

But, apparently, these are not in tension for at least some people.

Watching the Ninja Warrior clip...I think I have some sense of why anyone would watch sports, for the first time in my life.

The way those people can move is really impressive!

But the idea that the shape of one's life may be, in part, an unconscious treatment for mental flaws is a disquieting one.

I find this not at all disquieting? It seems like all of the badness comes from labeling your preferences a "mental flaw".

Is the implicit claim that you would have an overall better life if you pushed yourself to change or grow (another framing word) along this dimension? This is at least not obvious to me.