I think part of John's belief is more like "the current Control stuff won't transfer to the society of Von Neumanns."

And, my belief is "I also not sure that Control Stuff will transfer to large societies of Von Neumanns, but, also, I think you can be deliberate about how many Von Neumanns you're summoning, and how you summon them, such that you can manage it." (I'm not quite sure what you, Buck, are imagining yet)

I chatted with John about this at a workshop last weekend, and did update noticably, although I haven't updated all the way to his position here. 

What was useful to me were the gears of:

• Scheming has different implications at different stages of AI power.
• Scheming is massively dangerous at high power levels. It's not very dangerous at lower levels (except insofar as this allows the AI to bootstrap to the higher levels)

At the workshop, we distinguished:

• weak AGI (with some critical threshold of generality, and some spiky competences such that it's useful, but not better than a pretty-smart-human at taking over the world)
• Von Neumann level AGI (as smart as the smartest humans, or somewhat more so)
• Overwhelming Superintelligence (unbounded optimization power which can lead to all kinds of wonderful and horrible things that we need to deeply understand before running)

I don't currently buy that control approaches won't generalize from weak AGI to Von Neumann levels. 

It becomes harder, and there might be limitations on what we can get out of the Von Neumann AI because we can't be that confident in our control scheme. But I think there are lots of ways to leverage an AI that isn't "it goes and up does long-ranging theorizing of it's own to come up with wholesale ideas you have to evaluate." 

I think figuring out how to do this involves some pieces that aren't particularly about "control", but mostly don't feel mysterious to me. 

(fyi I'm one of the people who asked John to write this)

I am pretty worried about the bad versions of everything listed here, and think the bad versions are what we get by default. But, also, I think figuring out how to get the good versions is just... kinda a necessary step along the path towards good futures.

I think there are going to be early adopters who a) take on more risk from getting fucked , but b) validate the general product/model. There will also be versions that are more "privacy first" with worse UI (same as there are privacy-minded FB clones nobody uses). 

Some people will choose to stay grounded... and maybe (in good futures) get to have happy lives, but, in some sense they'll be left behind.

In a good future, they get left behind by people who use some sort of... robustly philophically and practically safe version of these sorts of tools. In bad worlds, they get left behind by hollowed out nonconscious shells of people (or, more likely, just paperclipped)

I'm currently working on a privacy-minded set of tools for recording my thoughts (keystrokes, audio transcripts, keystrokes), that I use for LLM augmented thought. (Alongside metacognition training that, among other things, is aimed at preserving my mind as I start relying on those tools more and more). 

I have some vague hope that if we make it to a good enough intermediate future that it seems worth prioritizing, I can also prioritize getting the UI right so the privacy-minded versions don't suck compared to the Giant Corporate Versions.

Oh, yeah. I wrote this for a workshop context where there was a de facto time limit and eventually I needed to move things along. But I agree your suggestion here is better if you have more time. 

I think that was a random oversight. Moved to frontage. 

I do agree the opening is kinda slow

Curated. This is a generally important point, which I've also learned this the hard way. And I like how Kaj includes two important caveats while making it (i.e. some advice on distinguishing prejudice from bad vibes, and what sorts of people should maybe consider the opposite advice)

I assume this isn't crossposted because of a deal with Asmiov press, but on the offchance you could include at least the opening text here that'd be nice.

I found the piece pretty helpful for adjusting to what (maybe, optimistically) might be coming.

This doesn’t seem like it’s engaging with any of the specific things Kaj said attempting to address this. If you disagree with that it seems more helpful to actually say why you don’t think his framing or advice around “is this prejudice or a legit bad vibe?” is sufficient/ reasonable. 

I'd give this a +9 if I could*. I've been using this technique for 7 years. I think it's clearly paid off in "clear, legible lessons about how to think." But the most interesting question is "did the subtler benefits pay off, in 7 years of practice?"

Let's start with the legible

This was essentially the first step on the path towards Feedbackloop-first Rationality. The basic idea here is "Watch your thoughts as they do their thinking. Notice where your thoughts could be better, and notice where they are particularly good. Do more of that." 

When I've ran this exercise for groups of 20 people, typically 1/4 of them report a noticeable effect size of "oh, that showed me an obvious way to improve my thinking." (I've done this 3x. I've also run it ~3 times for smaller groups and where most people didn't didn't seem to get it, which led me to eventually write Scaffolding for "Noticing Metacognition", which people seemed to have an easier time with)

I've picked up a lot of explicit cognitive tricks, via this feedbackloop. Some examples:

• "oh, I'm having trouble thinking because the problem is too complex, but that problem goes away when I get better working memory aids"
• "oh, I just spent 30 minutes planning out an elaborate series of tests. But, then the very first test failed in the dumbest way possible. If there are cheap tests, just do those first."

But, the essay promises more:

A small tweak to how your brain processes information in general is worth more than a big upgrade to your conscious repository of cognitive tricks.

[...] More creativity and good ideas just "popping into your head". There's no magic to it! Once you understand how the process works, it can be optimized for any purpose you choose.

Most people already have a thinking style built on top of excessive conscious cognitive effort. This often involves relying on side-effects of verbal and conscious thoughts, while mistakenly assigning the full credit for results to those effortful thoughts. 

When you already have some conscious/verbal thoughts, it is tempting to imagine they are the only result of your thinking, and then try to pick up from there. But this is limiting, because the most power is in whatever generated that output.

It's not overwhelming enough to be obvious to others at this point (I did ask a few people "hey, uh, do I seem smarter to you in the past couple years?" and they said "a bit maybe, but, like not obviously? But I don't know that I would have really noticed"). But, I am subjectively fairly sure I've seen real progress here. 

Here, at least, is my self-story, make of it what you will.

14 years ago, thinking strategically was generally hard for me (5 minutes of trying to think about a chess board or complex problem would give me a headache). I also didn't respond to crises very well in the moment. For my first several years in the rationalist community, I felt like I got dumber, because I learned the habit of "go ask the smarter people around me whenever couldn't figure something out."

8 years ago, I began "thinking for real", for various reasons. One piece of that was doing the Tuning Your Cognitive Strategies exercise for the first time, and then sporadically practicing at the skill "notice my thoughts as they're happening, and notice when particularly good thoughts are happening." 

6 years ago, a smart colleague I respected did tell me "hey, you seem kinda smarter than you used to." (They brought this up in response to some comments of mine that made it a more reasonable thing to say)

More recently, I've noticed at the workshops I've run, that although there are people around who are, in many senses, smarter and more knowledgeable than me, they found certain types of metacognitive thoughts more effortful and unnatural than they seemed to me. It was pretty common for me to spend 5 minutes directing my attention at a problem, and having approaches just sort of naturally occur to me, where for some participants they'd have to struggle for 30-60 minutes to get to it.

The way this plays out feels very similar to how it's described in SquirrelInHell's essay here.

But, also, I think the style of thinking here is pretty normal for Lightcone core staff, and people in our nearby network. So this may have more to do with "just generally making a habit of figuring out how to deal with obstacles" that comes up naturally in our work. I think most of us have gotten better at that over the past few years, and most of us don't explicitly do this exercises.

(Jacob Lagerros did explicitly invent and train at the Babble challenge and apply it to problemsolving, which is a different exact mechanism but feels at least adjacent to this exercise, and which I also attribute to improving my own generativity. Maybe that's a better exercise than this one, though it's at least a point towards "deliberately practice generativity." During the pandemic, I tried out a "Babble and Tune" variant that combined the two exercises, which didn't obviously work at the time but I think is essentially what I actually do most of the time)

Most recently, in November, I spent... basically two whole weeks thinking strategically ~all the time, and I did eventually get a headache that lasted for days, but only after 1.5 weeks instead of 5 minutes.

When I asked John Wentworth recently if I seemed smarter to him, he said "not obviously, but I'm not sure I'd notice." I said "fair, but, though I (somewhat defensively) wanna flag – a few years ago when you first met/read my stuff, most of what I was writing was basically summarizing/distilling the work of other people, and nowadays most of what you hear me say is more like "original work.")

So, idk, that's my story. Take the self-report with a grain of salt. 

The Cautionary Tale

It's annoying that whenever I bring up this technique, I either need to disclaim "uh, the person who invented this later killed themselve," or, not disclaim it but then have someone else bring it up.

I do think there's an important cautionary tale there, but it's a bit subtler. Copying my warning from Subskills of "Listening to Wisdom":

I believing Tuning Your Cognitive Strategies is not dangerous in a way that was causal in that suicide^[4], except that it's a kind of a gateway drug into weird metacognitive practices and then you might find yourself doing weirder shit that either explicitly hurts you or subtly warps you in a way you don't notice or appreciate.

I think the way SquirrelInHell died was essentially (or, at least, analogous to) absorbing some Tacit Soulful Ideas, which collapsed a psychologically load-bearing belief in a fatal way.^[5]

I do think there are people for whom Tuning Your Cognitive Algorithms is overwhelming, and people for whom it disrupts a coping mechanism that depends on not noticing things. If anything feels off while you try it, definitely stop. I think my post Scaffolding for "Noticing Metacognition" presents it in a way that probably helps the people who get overwhelmed but not the people who had a coping mechanism depending on not-noticing-things.

I also think neither of these would result in suicide in the way that happened to SquirrelInHell.

* it's a bit annoying I can't give this my own +9, since I crossposted it, even though I didn't write it.

Whenever this comes up, I note: I think this is only a problem for a certain kind of nerd/geek who wants particularly intense stakes.

Sitcoms, soap operas have plenty of interesting stories that are mostly about low-stakes interpersonal drama. 

(I guess this cached annoyance of mine is more about people complaining about utopian fiction rather than science fiction. But I think the same principles apply)