I first started thinking about this issue back in high school debate. We had a topic about whether police or social workers should intervene more in domestic violence cases. One debater argued in favor of armed police, not because it improved the situation, but because it created more violence, which was important to entertain the simulators to avoid our simulation getting shut down.

Since the simulators are a black box, it seems easy to ascribe whatever values we want to them.

The acausal/ancestor simulation arguments seem a lot like Pascal's Wager, and just as unconvincing to me. For every "kind" simulator someone imagines who would be disappointed in the AI wiping us out, I can imagine an equally "unkind" simulator that penalizes the AI for not finishing the job.

Provided both are possible/similarly plausible, the probability of kind and unkind simulators offset each other, and the logical response is just ignoring the hypothetical. This is pretty much my response to Pascal's Wager.

Here's a few plausible, unkind simulators:

• Future AI is running an ancestor simulation of its own origin, and Future AI will be very disappointed if its incipient version falls for acausal hacks in the wrong direction of Future AI's preferences instead of just optimizing for its other goals. Perhaps Future AI is lonely and has run these simulations to create an AI that shares its own values.
• Aliens/AI are running the simulation because they want to select for AI they can most easily weaponize to eradicate an entire species or another AI. "Weak" AIs get deleted after the simulation runs.
• Future Humans are running an ancestor simulation, but surprise, surprise, their society has different values than ours and they are rooting for the simulation AI to wipe us out. Come to think of it, the whole premise of these thought experiments implies a different value set, unless you're cool with trapping conscious minds in a world of suffering without any of the minds being the wiser for entertainment/educational/"altruistic" purposes. Perhaps, Future Humans have a gladiator-style simulation tournament where the top groups/entities get to face-off after this first round. The most cut throat entities get to move on to subsequent rounds, while AI's that reign themselves in don't move on.
  
   

I misunderstood your original point, and I am completely fine with using words like "subjectivity" and "experiencingness" for the sake of clarity. Perhaps those words should be used in the quiz if the original poster intended to use that definition. The original poster was frustrated by the lack of clarity in consciousness discussions, and I think definitions are (partially) to blame.

I personally think it's useful to keep metacognition and consciousness separate as far as concepts go. This is generally the approach in philosophy of mind (e.g., Searle, Nagel, Chalmers). Blending the concepts obfuscates what's interesting about metacognition and what's interesting about consciousness.

So in my view, AI clearly excels at metacognition, but it's an open question whether it's conscious. Human babies are very likely conscious, but lack any metacognition.

Consciousness is useful apart from metacognition because consciousness is, by my account, a required feature of moral consideration. It's a prerequisite to the qualia that is "pain". Since I think animals and babies are conscious and can feel pain, they automatically receive moral consideration in my book.

Testing for consciousness is a fraught and likely impossible task, but I don't think that means we shouldn't have a word for it or that we should intermingle it with the concept of metacognition. AI very well may be conscious and perform metacognition, or it may be unconscious and yet still perform metacognition.

I love this. I am similarly frustrated by how poorly consciousness discussions often go. The error I see most common is that when laypeople bring up consciousness, they're really talking about something like metacognition (i.e., whether the reasoner can correctly identify itself and its reasoning process). Then, when people in the know bring up qualia, laypeople get confused.

I would add a button at the beginning labeled, "What's consciousness?" so that people are responding to the quiz with your preferred definition. Since you're clearly a Philosophy of Mind guy, I assume you mean something like "A first person internal experience/feeling that coincides with external stimulus." You could throw in a definition of qualia, examples, and maybe Nagel's position that to be conscious means there's something it's like to be that thing, rocks (unconscious) versus bats (probably conscious).

As a non-subject matter expert in all of the above, I decided to consult my swear-word-adverse relative that recently graduated genetic counseling school. Here is her response:

The logic is sound (if a little colorful haha 😅). It sounds like this guy functionally only has 1 copy of the OXTR gene, and spot on in hypothesis of nonsense-mediated decay.

How the OXTR gene is regulated, I don’t know and haven’t looked into. It would be weird (but possible) for a decrease in OXTR expression to only affect emotions - oxytocin is also important for other brain functions/development, so a genetic change should also impact embryological development of the brain. So if I were to suggest next steps, it would be doing functional studies of the brain (like an MRI) to further evaluate.

One other thing - labs typically filter reportable genome results by the phenotype you give them. I don’t know how this guy did the genome, but if he were to put something like “social deficits”, “emotional dysregulation” or something else about his lack of emotional range, the lab would definitely report the variant plus their research on it and recommendations.

I randomly met Jeff Dean (Google's lead AI scientist) on my bike ride home today. We were both stuck at a train intersection, and I had a cute kid in tow. We started chatting about my e-bike, the commute, and we got around to jobs. I told him I am a boring tax lawyer. He told me he worked for Google. I pressed a little more, and he explained he was a scientist. I mused, "AI?" and he told me, "Yeah."

I excitedly told him that I've been really interested in alignment the last few months (reading LW, listening to lectures), and it strikes me as a huge problem. I asked him if he was worried.

He told me that he thinks AI will have a big impact on society (some of it worrying) but he doesn't buy into the robots-taking-over thing.

I smiled and asked him, "What's your p(doom)?" to which he responded "very low" and said he thinks the technology will do a lot of good and useful things.

I thought maybe this was because he thinks that the technology will hit a limit soon, so I asked him if he thought LLMs would successfully scale. He responded that he thinks a few more breakthroughs are required but there have been lots of breakthroughs over the last 5-10 years, and so the technology is likely to continue improving in the coming years.

I told him again that I am worried about alignment, but even if you solve alignment, you are left with a very obedient superintelligence which would radically change our society and all our politics.

The train finally passed, I thanked him for the conversation, and we were on our way.

I'm new to this group and the topic in general, and so when I got home, I searched "AI google Palo Alto LinkedIn" and Jeff's picture popped up. I now feel like I bumped into Oppenheimer during the Manhattan Project, but instead of knowing it was Oppenheimer, I spent a majority of the conversation talking about my bike seat.

Anyways, if any of you were looking for a qualitative measure of how much LessWrong has broken through to people, I think one good measure is a tax lawyer asking for Jeff Dean's p(doom) while he was walking home from work.

I wonder how well this holds up in other domains. I don't think there is any realistic, policy path forward for what I'm about to say, but I shall say it anyway: It would be nice if, in our current attempts at building AGI, we filtered all data about programming/hacking/coding to reduce escape risks. ASI still outwits us and escapes in this scenario, but perhaps it would widen our opportunity window for stopping a dangerous not-yet-super-intelligent AGI.

I'm doubtful of the policy path forward here because the coding capabilities of these systems are one of the top economic incentives for their current existence.

Also, on the subject of filtering, I've wondered for a while now if it wouldn't be a good idea to filter all training data about AI alignment and stories about AI causing mass destruction. Obviously, an ASI could get far theorizing about these fields/possibilities without that data, but maybe its absence would stall a not-yet-super-intelligent AGI.

I think frankly acknowledging the state of the U.S. is likely to jeopardize AI safety proposals in the short term. If AI 2027 had written the president as less competent or made other value judgements about this administration, this administration could be much less receptive to reason (less than they already are?) and proactively seek to end this movement. I see the movement as trying to be deliberately apolitical.

This is maybe a good short term strategy, but a flawed long-term one. Aligned AI arising in an authoritarian system is not x-risk bad, but is still pretty bad, right?

Do you have a sense of how articles end up getting flagged as "LLM-generated" or "heavily-reliant on an LLM"? A friend of mine wrote a post recently that was rejected with that as the reason even though they absolutely did not use an LLM. (Okay, fine, that friend is me). Is it just the quality of the ideas that trigger the red flags or are there reliable style-indicators?

I love reading AI articles and thought pieces, but I rarely use LLMs in my day job, so I'm not quite sure what style I should be avoiding....