There's a tiny possibility he may have influenced my thinking. I did spend 6 months editing him, among others for a documentary.

That looks interesting, will read :) Thanks.

What an insightful post!

I have difficulty judging how likely that is, but the odds will improve if semi-wise humans keep getting input from their increasingly wise AGIs.

I think we're on the same page here, positing that AGI could actually help to improve alignment—if we give it that task. I really like one of your fundamental instructions being to ask about potential issues with alignment.

And on the topic of dishing out tasks, I agree that pushing the industry toward Instruction Following is an ideal path, and I think there will be a great deal of consumer demand for this sort of product. A friend of mine has mentioned this as the no-brainer approach to AI safety and even a reason what AI safety isn't actually that big a deal... I realise you're not making this claim in the same way.

My concern regarding this is that the industry is ultimately going to follow demand and as AI becomes more multi-faceted and capable, the market for digital companions, assistants and creative partners will incentivise the production of more human, more self-motivated agents (sovereign AGI) that generate ideas, art and conversation autonomously, even spontaneously.

Some will want a two-way partnership, rather than master-slave. This market will incentivise more self-training, self-play, even an analogue to dreaming / day-dreaming (all without a HITL). Whatever company enables this process for AI will gain market share in these areas. So, while Instruction Following AI will be safe, it won't necessarily satisfy consumer demand in the way that a more self-motivated and therefore less-corrigible AI would.

But I agree with you that moving forward in a piecemeal fashion with the control of an IF and DWIMAC approach gives us the best opportunity to learn and adapt. The concern about sovereign AGI probably needs to be addressed through governance (enforcing HITL, enforcing a controlled pace of development, and being vigilant about the run-away potential of self-motivated agents) but it does also bring Value Alignment back into the picture. I think you do a great job of outlining how ideal an IF development path is, which should make everyone suspicious if development starts moving in a different direction. 

Do you think it will be possible to create an AGI that is fundamentally Instruction Following that could satisfy the market for the human-like interaction some of the market will demand?

I apologise if you've, in some way I've not recognised, already addressed this question, there were a lot of very interesting links in your post, not all of which I could be entirely sure I grokked adequately.

Thanks for your comments, I look forward to reading more of your work.

Thanks, very astute point.

Yes, the individual and the collected are tightly coupled with short-term and long-term goals, which exist within individuals too. I think it's interesting to think of yourself as a city, where you need to make systemic changes sometimes to enable individual flourishing.

I really think there is something to making alignment the actual goal of AI—but in a way where the paradoxical nature of alignment is acknowledged, so the AI is not looking for a "final solution" but is rather measuring the success of various strategies in lowering society's (to return to the metaphor of the individual) cognitive dissonance.

Thanks Seth, great points.

Would I like your drawings less if you had conceived them in detail, but not directed the brushstrokes with your own hands and brain?

I guess, for centuries revered artists have been directing apprentices to render works for them, AI is not a significant departure from this practice, and I think you're right—there is still a connection with an artist even if their role was to prompt for an emotion, and then, in their selection, recognise that emotion when the AI successfully captured it.

This is making me hope that we see more detailed accounts of the creative process

This is really interesting. As your rightly state, making their intentions explicit is not a common practice for artists, my cynicism again tells me this could be c) part of the inflationary imperative of artists to play on the allure of ineffable qualities in art to make something relatively shallow seem deep and meaningful (and therefore valuable).

Imagine a space where truly deep thoughts, and profound messages well put could generate art that really connects people—not because it means something different to every person but because it evokes the very same thing, common humanity, among unique individuals. Pretty lofty.

Thanks again for your kind words and insight.

Well, yes, good point—people consume art for all sorts of reasons.

Though I wasn't meaning to say that anyone consciously looks at an artwork with the intention of connecting with the artist, only that it's an implied prerequisite, as in, if we're impressed by the skill, we're impressed because we have a sense of how difficult that would be for a human (being a human ourselves) or if we think the work has captured an emotion we might implicitly assume that the artist recognised that same emotion in creating the work. These features of the art-consumption experience are largely absent in AI art, when we are pretty certain that the "artist" has no conscious experience.

But, yes, I take your point, and people can appreciate AI art for many reasons besides.

That was fascinating, and well written, thanks for sharing. There I thought eventually having my wisdom that out would be a hassle, now I’m looking forward to it.

Interesting the use of the word “fascinating” as that carries with it some sense of an emotional experience, some excitement. But perhaps, like a language model can be used as simple hyperbole. Also the enthusiasm noted, suggesting that enthusiasm and conscious experience can be decoupled.

Thanks for sharing, definitely one of those experiments that will inform my thinking next time I’m thinking about AI sentience.

Hi Seth,

Thanks for your kind words. It's funny, I think I naturally write in a longer more convoluted style, but have worked hard to make my writing accessible and short—nice to know the effort pays off.

The cartoons are drawn with an Apple Pencil on an iPad Pro using Procreate (the studio pen is great for cartooning if you're really interested). I set up a big canvas 1000px wide and about 5000px high, then go about drawing all of them top to bottom. Then I export to photoshop, crop and export to png with a transparent background so that whatever colour the page is shows through. Those I've used here on LW are screenshots from the blog itself as the image backgrounds don't work well on white or black (the only options here—my site is generally a pastel blue). I've explained in another post why I'm keeping my crappy drawings in the face of the generative AI revolution.

Thanks for the extra info around terms like "coordination", good to know. I actually mention Moloch in part 2 and have written a series on Moloch, funny you use the word "thorny", as the cartoon characters I use for that series are called "Thorny Devils" (Moloch Horridus).