We get like 10-20 new users a day who write a post describing themselves as a case-study of having discovered an emergent, recursive process while talking to LLMs. The writing generally looks AI generated. The evidence usually looks like, a sort of standard "prompt LLM into roleplaying an emergently aware AI".

It'd be kinda nice if there was a canonical post specifically talking them out of their delusional state. 

If anyone feels like taking a stab at that, you can look at the Rejected Section (https://www.lesswrong.com/moderation#rejected-posts) to see what sort of stuff they usually write.

Reading through Backdoors as an analogy for deceptive alignment prompted me to think about a LW feature I might be interested in. I don't have much math background, and have always found it very effortful to parse math-heavy posts. I expect there are other people in a similar boat.

In modern programming IDEs it's common to have hoverovers for functions and variables, and I think it's sort of crazy that we don't have that for math. So, I'm considering a LessWrong feature that:

• takes in a post (i.e. when you save or go to publish a draft)
• identifies the LaTeX terms in the post
• creates a glossary for what each term means. (This should probably require confirmation by the author)
• makes a hoverover for each term so when you mouseover it reminds you.

On "Backdoors", I asked the LessWrong-integrated LLM: "what do the Latex terms here mean"?

It replied :

The LaTeX symbols in this passage represent mathematical notations. Let me explain each of them:

1. $F$ : This represents a class of functions. The curly F denotes that it's a set or collection of functions.
2. $f\in F$ : This means that $f$ is a function that belongs to (is an element of) the class $F$.
3. $f^{\ast }$ : The asterisk superscript typ

The “prompt shut down” clause seemed like one of the more important clauses in the SB 1047 bill. I was surprised other people I talked to didn't think seem to think it mattered that much, and wanted to argue/hear-arguments about it.

The clauses says AI developers, and compute-cluster operators, are required to have a plan for promptly shutting down large AI models.

People's objections were usually:

"It's not actually that hard to turn off an AI – it's maybe a few hours of running around pulling plugs out of server racks, and it's not like we're that likely to be in the sort of hard takeoff scenario where the differences in a couple hours of manually turning it off will make the difference."

I'm not sure if this is actually true, but, assuming it's true, it still seems to me like the shutdown clause is the one of the more uncomplicatedly-good parts of the bill.

Some reasons:

1. I think the ultimate end game for AI governance will require being able to quickly notice and shut down rogue AIs. That's what it means for the acute risk period to end. 

2. In the more nearterm, I expect the situation where we need to stop running an AI to be fairly murky. Shutting down an AI is going to be ve... (read more)

I feel so happy that "what's your crux?" / "is that cruxy" is common parlance on LW now, it is a meaningful improvement over the prior discourse. Thank you CFAR and whoever was part of the generation story of that.

My Current Metacognitive Engine

Someday I might work this into a nicer top-level post, but for now, here's the summary of the cognitive habits I try to maintain (and reasonably succeed at maintaining). Some of these are simple TAPs, some of them are more like mindsets.

• Twice a day, asking “what is the most important thing I could be working on and why aren’t I on track to deal with it?”
  • you probably want a more specific question (“important thing” is too vague). Three example specific questions (but, don’t be a slave to any specific operationalization)
    • what is the most important uncertainty I could be reducing, and how can I reduce it fastest?
    • what’s the most important resource bottleneck I can gain, or contribute to the ecosystem, and would gain me that resource the fastest?
    • what’s the most important goal I’m backchaining from?
• Have a mechanism to iterate on your habits that you use every day, and frequently update in response to new information
  • for me, this is daily prompts and weekly prompts, which are:
    • optimized for being the efficient metacognition I obviously want to do each day
    • include one skill that I want to level up in, that I can do in the morning as part of the meta-orienting (su

I want to be able to talk about the tribal-ish dynamics in how LW debates AI (which feels indeed pretty tribal and bad on multiple sides).

A thing that feels tricky about this is that talking about it in any reasonable concise way involves, well, grouping people into groups, which is sort of playing into the very tribal dynamic I'd like us to back out of. 

If we weren't so knee-deep in the problem, it'd seem plausible that the right move is just "try to be the non-tribal conversation you want to exist in the world." But, we seem pretty knee-deep in into it and a few marginal reasonable conversations aren't going to solve the problem.

My default plan is "just talk about the groups, add a couple caveats about it", which seems better to me than not-doing-that. But, I do wish I had a better option and curious for people's takes.

Motif coming up for me: a lot of skill ceilings are much higher than you might think, and worth investing in.

Some skills that you can be way better at:

• Listening to people, and hearing what they're actually trying to say, and gaining value from it
• Noticing subtle things that are important. You can learn to notice like 5 different things happening inside you or around you, that occured in <1 second.
• Being concrete, in ways that help you resolve confusion and gain momentum on solving problems.
• Each stage of OODA Looping is quite deep
  • (i.e. "Observe", "Orient", "Decide", and "Act" each have a lot of deep subskills. The depth of "Noticing" is a subset of the overall set of "Observation" skills")

There was a particular mistake I made over in this thread. Noticing the mistake didn't change my overall position (and also my overall position was even weirder than I think people thought it was). But, seemed worth noting somewhere.

I think most folk morality (or at least my own folk morality), generally has the following crimes in ascending order of badness:

• Lying
• Stealing
• Killing
• Torturing people to death (I'm not sure if torture-without-death is generally considered better/worse/about-the-same-as killing)

But this is the conflation of a few different things. One axis I was ignoring was "morality as coordination tool" vs "morality as 'doing the right thing because I think it's right'." And these are actually quite different. And, importantly, you don't get to spend many resources on morality-as-doing-the-right-thing unless you have a solid foundation of the morality-as-coordination-tool.

There's actually a 4x3 matrix you can plot lying/stealing/killing/torture-killing into which are:

• harming the ingroup
• harming the outgroup (who you may benefit from trading with)
• harming powerless people who don't have the ability to trade or col

There's a skill of "quickly operationalizing a prediction, about a question that is cruxy for your decisionmaking."

And, it's dramatically better to be very fluent at this skill, rather than "merely pretty okay at it."

Fluency means you can actually use it day-to-day to help with whatever work is important to you. Day-to-day usage means you can actually get calibrated re: predictions in whatever domains you care about. Calibration means that your intuitions will be good, and _you'll know they're good_.

Fluency means you can do it _while you're in the middle of your thought process_, and then return to your thought process, rather than awkwardly bolting it on at the end.

I find this useful at multiple levels-of-strategy. i.e. for big picture 6 month planning, as well as for "what do I do in the next hour."

I'm working on this as a full blogpost but figured I would start getting pieces of it out here for now.

A lot of this skill is building off on CFAR's "inner simulator" framing. Andrew Critch recently framed this to me as "using your System 2 (conscious, deliberate intelligence) to generate questions for your System 1 (fast intuition) to answer." (Whereas previously, he'd known System 1 ... (read more)

Something struck me recently, as I watched Kubo, and Coco - two animated movies that both deal with death, and highlight music and storytelling as mechanisms by which we can preserve people after they die.

Kubo begins "Don't blink - if you blink for even an instant, if you a miss a single thing, our hero will perish." This is not because there is something "important" that happens quickly that you might miss. Maybe there is, but it's not the point. The point is that Kubo is telling a story about people. Those people are now dead. And insofar as those people are able to be kept alive, it is by preserving as much of their personhood as possible - by remembering as much as possible from their life.

This is generally how I think about death.

Cryonics is an attempt at the ultimate form of preserving someone's pattern forever, but in a world pre-cryonics, the best you can reasonably hope for is for people to preserve you so thoroughly in story that a young person from the next generation can hear the story, and palpably feel the underlying character, rich with inner life. Can see the person so clearly that he or she comes to live inside them.

Realistical... (read more)