Three Fables of Magical Girls and Longtermism

Before I opened this I thought it was another GPT query lol

I also recommend To The Stars, a PMMM fanfic set in the far future that inspired dath ilan’s Governance (warning: ~850k words and incomplete): https://archiveofourown.org/works/777002/

[-]jayterwahl3y50

Where did you hear that TTS inspired dath ilan's Governance?

[-]LawrenceC3y60

https://twitter.com/ESYudkowsky/status/1590113143778578432

Rereading a bit of Hieronym's PMMM fanfic "To The Stars" and noticing how much my picture of dath ilan's attempt at competent government was influenced / inspired by Governance there, including the word itself. https://archiveofourown.org/works/777002/chapters/1461984

EDIT: whoops, Tetraspace got there first.

[-]Tetraspace3y21

@ESYudkowsky on Twitter:

Rereading a bit of Hieronym's PMMM fanfic "To The Stars" and noticing how much my picture of dath ilan's attempt at competent government was influenced / inspired by Governance there, including the word itself.
https://archiveofourown.org/works/777002/chapters/1461984

[-]Julian Bradshaw3y40

To the Stars is an interesting universe in which AI alignment was solved (or, perhaps, made possible at all) via magical girl wish! Quoting (not really a spoiler since this is centuries in the past of the main story):

It'd be nice if, like Kekulé, I could claim to have some neat story, about a dream and some snake eating itself, but mine was more prosaic than that.
I had heard about the Pretoria Scandal, of course, on the day the news broke. To me, it was profoundly disturbing, enough that I ended up laying awake the whole night thinking about it.
It was an embarrassment and a shame that we had been building these intelligences, putting them in control of our machines, with no way to make sure that they would be friendly. It got people killed, and that machine, to its dying day, could never be made to understand what it had done wrong. Oh, it understood that we would disapprove, of course, but it never understood why.
As roboticists, as computer scientists, we had to do better. They had movies, back then, about an AI going rogue and slaughtering millions, and we couldn't guarantee it wouldn't happen. We couldn't. We were just tinkerers, following recipes that had magically worked before, with no understanding of why, or even how to improve the abysmal success rate.
I called a lab meeting the next day, but of course sitting around talking about it one more time didn't help at all. People had been working on the problem for centuries, and one lab discussion wasn't going to perform miracles.
That night, I stayed in late, pouring over the datasets with Laplace, [the lab AI,] all those countless AI memory dumps and activity traces, trying to find a pattern: something, anything, so that at least we could understand what made them tick.
Maybe it was the ten or something cups of coffee; I don't know. It was like out of a fairy tale, you know? The very day after Pretoria, no one else in the lab, just me and Laplace talking, and a giant beaker of coffee, and all at once, I saw it. Laplace thought I was going crazy, I was ranting so much. It was so simple!¹
Except it wasn't, of course. It was another year of hard work, slogging through it, trying to explain it properly, make sure we saw all the angles…
And I feel I must say here that it is an absolute travesty that the ACM does not recognize sentient machines as possible award recipients.² Laplace deserves that award as much as I do. It was the one that dug through and analyzed everything, and talked me through what I needed to know, did all the hard grunt work, churning away through the night for years and years. I mean, come on, it's the Turing Award!
The MSY has confirmed that the timing of this insight corresponds strongly with a wish made on the same day. The contractee has requested that she remain anonymous.
The ACM removed this restriction in 2148.
— Interview with Vladimir Volokhov, Turing Award Recipient, 2146.

(The actual content of the alignment solution is elsewhere described to be something like a chain of AIs designing AIs via a mathematically-provable error-correcting framework, continuing until the output stabilized—for what it's worth.)

[-]Raemon3y40

This is as good a place as any to note that The Circle, the EA song about the expanding circle of concern, has gotten an music video with footage set to Madoka footage:

[-]antanaclasis3y21

Thanks for writing this nice article. Also thanks for the “Qualia the Purple” recommendation. I’ve read it now and it really is great.

In the spirit of paying it forward, I can recommend https://imagakblog.wordpress.com/2018/07/18/suspended-in-dreams-on-the-mitakihara-loopline-a-nietzschean-reading-of-madoka-magica-rebellion-story/ as a nice analysis of themes in PMMM.

[-]Emrik3y1-2

Still the only anime with what at least half-passes for a good ending. Food for thought, thanks! 👍

[-]ponkaloupe3y10

just don’t watch the subsequent film, which completely unravels the original ending 😢

[-]Ulisse Mini3y12

unpopular opinion: I like the ending of the subsequent film

IMO it's a natural continuation for Homura. After spending decades of subjective time trying to save someone would you really let them go like that? Homura isn't an altruist, she doesn't care about the lifetime of the universe - she just wants Madoka.

[-]LawrenceC3y20

I know that everyone says it's an "unpopular opinion" but surely it's the dominant opinion by now.

^{^}

Though it isn't spelled out in the show, humans appear to be the only species that has feelings, so depending on whether you're a positive or negative utilitarian, a universe full of emotionless beings may or may not be a compelling vision for you.

LESSWRONG
LW

LESSWRONG
LW

33

Three Fables of Magical Girls and Longtermism

33

33

Kyubey Shuts Up And Multiplies

Homura has Something to Protect

Magical Girls are mesa-optimizers

In Conclusion