With such a disclaimer, I think I would have been less confused but still not positive on the text. Maybe I'd be less negative? Neutral, even? A lack of disclaimer is not really the core of it for me though.

I have strange tastes in fiction, I think, and normally after reading something that anti-resonated with me so much, I'd just stop reading. But, I read to the end because a bunch of upvotes signaled to me that I might otherwise miss out on something later in the text.

I live in the Bay Area and work in tech and follow AI news. For me, I already knew that there are traits I don't like in others, and had approximate models of their frequency in and correlations to the AI scene. Each character struck me as plausible and not really meaningfully more concrete to me. Concrete in the sense of "Oh, this behavior is warning sign that this is a morally bankrupt person I should stay away from" or "I might accidentally be falling into this pattern if I find this character relatable in some way." I can definitely see this being useful to others who do not live in the bay area/work in tech/do not follow AI news, but I mostly expected people on lesswrong to match enough of that description that they wouldn't find themselves learning from this "concretely describing a vibe or something." And indeed, some of the people liked it for reasons other than that, which I failed to predict!

A big part of it for me is that I just cannot stand the first person narration of this main character. An example line, pulled at random: "But regardless, she makes very good coffee and is very nice and beautiful which sometimes amounts to the same thing." I find this thoroughly jarring. I read it as the character's internal monologue, but this is just so thoroughly outside the domain of what would go in my own internal monologue it feels... rude? an empty attempt at caricature? a failure to understand what internal thoughts look like or otherwise an indication of a mind horribly alien to my own such that it bears no resemblance to real humans? I do find the characters unappealing, but this just seemed unnecessary to me.

As an update, I've now gotten a bunch of useful responses of why people liked this! I was worried people would interpret me as being needlessly negative, but it seems mostly not so. I think I'm still far from enjoying this myself, but I do think I've learned a good amount from the answers I got, and I got some interesting things to think about.

I just went and read that one and found it interesting, yes!

That said, even if I'm unlikely to get a satisfactory response, I still want to ask. I would like to be able to better predict what other people like/think. And sometimes, I've even found that understanding someone else can help find new dimensions to appreciate :)

I personally don't believe "writing quality" can be divorced from content, and if you shove a bunch of words together in a "masterful way" but don't say anything, I don't want to read that and therefore would call that bad writing.

Upon slightly more reflection, I think I can appreciate on an intellectual level the quality of the writing if the goal is to evoke disgust. People (myself included, on occasion) partake in Horror content, which is also traditionally a negative emotion. I haven't heard of a Revulsion genre before, so I didn't really consider that this might be a thing people pursue, but I would still be a little surprised if that was what most people got out of the post.

I would also be surprised if the source of all the upvotes was just that it is """high quality""" writing. I usually find LessWrong to be more focused on content, and I still want to know what other people see in this post.

I did not enjoy this. I did not feel like I got anything out of reading this.

However, this got curated and >500 karma, so I feel like I must be missing something. Can anyone inform me: Why did other people enjoy this? The best theory I can scrounge together is that this is "relatable" in some way to people in SF, like it conveys a vibe they are feeling?

If the goal is to evoke a sensation of disgust with the characters, then I guess you've succeeded for me. I already knew I would not like the sort of person described in this story, though, so I didn't feel like I learned anything as a result, but I could see how something like that could be useful for others. I essentially just felt disgust the entire time reading this.

Artificial General Intelligence, AGI, is an AI, that can do anything an individual human can do, (especially on economically productive metrics).

Artificial Superintelligence, ASI, is an AI, that can do much more than all of humanity working together.

These definitions wouldn't be suitable for a legal purpose, I imagine, in that they lack a "technical" precision. However, in my mind, there is a very big difference between the two, and an observer wouldn't be likely to mislabel a system as ASI when it is actually AGI, or vice versa.

Yet, in my mind, one of the biggest risks of AGI is that it is used to build ASI, which is why I still agree with your post.
 

I like most of this post, but:

AGI != ASI. Defining AGI, and only post ante-fining a company into oblivion that makes AGI may be enough to prevent the death of humanity. I would put pretty good odds on it being enough, as long as it was strongly enforced and detected.

I would still support regulation preventing AGI, I just want the terminology to be straight. ASI is the thing that IABIED.

I would love an excuse to go back and learn QFT. Looking forward to your QFT AI insights :D

Although as I note elsewhere I’m starting to have some ideas of how something with elements of this might have a chance of working.

I've missed where you discussed this. Does anyone have a link or can anyone expound?

I think the problem of "actually specifying to an AI to do something physical, in reality, like 'create a copy of strawberry down to the cellular but not molecular level', and not just manipulate its own sensors to believe it perceives itself achieving that even if it accomplishes real things in the world to do that" is a problem that is very deeply related to physics, and is almost certainly dependent on the physical laws of world more than some abstract disembodied notion of an agent.