All of Kaj_Sotala's Comments + Replies

I meant in the sense that there were quite a few different pieces of evidence presented in the post  (e.g. this was one index out of three mentioned), so just pointing out that one of them is weaker than implied doesn't affect the overall conclusion much.

6Raemon3d
Fwiw I still don't think it makes sense to call that a nitpick. Seems like a good thing to point out. (I agree it's not, like, a knockdown argument against the whole thing. But I think of nitpicks as things that aren't relevant to the central point of the post)

This is a nitpick that doesn't really affect the overall message of the post (which I upvoted), but:

The Economist's Democracy Index shows a sharp decline over the last decade:

That chart has a cut y-axis; the decline looks much less sharp in a graph that shows the full range:

This box from Wikipedia also suggests that the overall average going from 5.55 to 5.3 isn't that significant, as the whole scale is used and everything between 4 and 6 is considered to be within the same category of regime:

+1.

I'm a big fan of extrapolating trendlines, and I think the current trendlines are concerning. But when evaluating the likelihood that "most democratic Western countries will become fascist dictatorships", I'd say these trends point firmly against this being "the most likely overall outcome" in the next 10 years. (While still increasing my worry about this as a tail-risk, a longer-term phenomena, and as a more localized phenomena.)

If we extrapolate the graphs linearly, we get:

  • If we wait 10 years, we will have 5 fewer "free" countries and 7 more "non-free
... (read more)

In what sense is that a nitpick or something that doesn't affect the message? It's a substantial drag on the message, data that only supports the conclusion if you already have a prior that the conclusion is true.

Good point, but also according to Wikipedia "the index includes 167 countries and territories", so small changes in the average are plausibly meaningful.

Though note that this assumes that defenders are willing and capable of actually patching their systems. There are lots of people who are running outdated insecure versions of various pieces of software, product vendors with no process for patching their products (especially in the case of software embedded into physical products), etc. 

E.g.:

This report analyses 127 current routers for private use developed by seven different large vendors selling their products in Europe. An automated approach was used to check the router’s most recent firmware versi

... (read more)

I skimmed this a bit, but found it a little hard to follow along your notes without knowing what those first seven pages of the book were actually saying. I felt that in order to understand your thought process, I would first need to read your notes and reconstruct the argument the book was making from your responses to it, and then read the post a second time once I had some sense of what it was a reaction to. But that seems pretty error-prone; it would help a lot to have a summary of those seven pages before your notes.

I think there's also a third type of curiosity. The two that you mentioned sound goal-driven in a sense; you either want to understand how something works, or to fit it into a particular understanding of the world. And there's a sense in which the curiosity gets "finished" once it achieves that understanding, and then moves on.

That's in contrast to what I'd call "open" curiosity, which has no particular fixed goal or agenda that could be achieved. Instead it's something like, in each passing moment, being open and curious about what will happen next rather... (read more)

without any empirical data for or against.

I think there's plenty of empirical data, but there's disagreement over what counts as relevant evidence and how it should be interpreted. (E.g. Hanson and Yudkowsky both cited a number of different empirical observations in support of their respective positions, back during their debate.)

2shminux10d
Right. I'd think the only "empirical evidence" that counts is the one accepted by both sides as good evidence. I cannot think of any good examples.

Curated. I really liked this very clear discussion of bids and the development of trust. I also thought it had subtle but important points that aren't always mentioned, such as the way that trust built up via fulfilling all bids is fragile.

That's a fair question, I would guess that most of the people responding to those studies would still be in the habit of meditation.

On the other hand, I think that once people start hitting that intermediate range, they get to the point where meditative practices become automatic enough to happen in the middle of daily life. I myself only do a pretty limited amount of formally sitting down for a dedicated meditation session - my meditation app reports an average of 15 minutes per day over the last year - but I do feel like I do quite a bit of it at the sam... (read more)

3Anton Rodenhauser13d
Yes. I agree. Yet, there is this:  I've spent the last 3 years averaging around 3 hours of meditation a day. I've had many months with 6+ hours meditation a day. I had times when the boundary between formal practice in daily life was indeed very thin - in other words, it was relatively easy/automatic to be mindful more or less 24/7. Yet, in these rare times when I did not meditate at all for, say, 2 weeks (mostly because of health issues), I very quickly lose that ability to automatically be mindful throughout the day. I would guess that if I stopped meditating for a year and would not bother trying to be mindful throughout the day, my "mindfulness throughout the day" level would go back to basically zero.

And in my experience, insight into Emptiness, No-Self, etc., is transitory and not helpful anymore once you’ve stopped meditating huge amounts for a while.

Counterpoint: the research reviewed in Altered Traits suggested increasing permanent effects from meditation the longer you practice, with time spent on retreats being one significant factor.

... at the start of contemplative practice, little or nothing seems to change in us. After continued practice, we notice some changes in our way of being, but they come and go. Finally, as practice stabilizes, t

... (read more)
3Anton Rodenhauser15d
Fascinating! Really cool stuff! Thanks for sharing. Okay, I concede! Amassing many hours of just meditation on and off retreats over many years is definitely not "useless". Some effects definitely persist! That is actually also my experience with 5000+ hours of meditation and many retreats. I guess my key point is that those changes are overrated - especially given how much effort they take, and that in general there are far more effective ways to reach very similar goals! But there are some important exceptions to this. If you for example do manage to get enlightened and stabilize that state, that's just absolutely amazing, and no amount of ordinary therapeutic progress will ever get you the kind of beauty and certain mental superpowers that come with that.

I haven't tried doing an emotional work retreat as described here, but I endorse the general idea that most people will get more of the thing they want out of a combination of meditation + emotional work practices rather than meditation alone. Or if they had to choose just one, they'd probably be better off with the emotional practices rather than meditation.

in particular, we don't typically think of freedom as a property of relationships, but rather a property of individuals.

How about "spaciousness" (as in the relationship giving both individuals the space to move/act as they prefer) instead of freedom/trust?

4Richard_Ngo16d
Mmm, I still prefer trust I think. Spaciousness gives me connotations of... well, distance, and separation. In some sense my relationship with almost everyone in the world is spacious. The thing that's special about some relationships is that they have both spaciousness and intensity, which to me feels well-described by "trust".

Some notable/famous signatories that I noted: Geoffrey Hinton, Yoshua Bengio, Demis Hassabis (DeepMind CEO), Sam Altman (OpenAI CEO), Dario Amodei (Anthropic CEO), Stuart Russell, Peter Norvig, Eric Horvitz (Chief Scientific Officer at Microsoft), David Chalmers, Daniel Dennett, Bruce Schneier, Andy Clark (the guy who wrote Surfing Uncertainty), Emad Mostaque (Stability AI CEO), Lex Friedman, Sam Harris.

Edited to add: a more detailed listing from this post:

Signatories include notable philosophers, ethicists, legal scholars, economists, physicists, politica

... (read more)

Bruce Schneier has posted something like a retraction on his blog, saying he focused on the comparisons to pandemics and nuclear war and not on the word "extinction".

4Leon Lang15d
https://twitter.com/ai_risks/status/1664323278796898306?s=46&t=umU0Z29c0UEkNxkJx-0kaQ [https://twitter.com/ai_risks/status/1664323278796898306?s=46&t=umU0Z29c0UEkNxkJx-0kaQ] Apparently Bill Gates signed. Stating the obvious: Do we expect that Bill Gates will donate money to prevent the extinction from AI?
7Joel Burget18d
Though the statement doesn't say much the list of signatories is impressively comprehensive. The only conspicuously missing names that immediately come to mind are Dean and LeCun (I don't know if they were asked to sign).

Relevant: Goh et al. finding multimodal neurons (ones responding to the same subject in photographs, drawings, and images of their name) in the CLIP image model, including ones for Spiderman, USA, Donald Trump, Catholicism, teenage, anime, birthdays, Minecraft, Nike, and others.

To caption images on the Internet, humans rely on cultural knowledge. If you try captioning the popular images of a foreign place, you’ll quickly find your object and scene recognition skills aren't enough. You can't caption photos at a stadium without recognizing the sport, and you

... (read more)

Many commenters seem to be reading this post as implying something like slavery and violence being good or at least morally okay. Which is weird, since I didn't get that impression - especially since the poster explicitly says they don't support slavery and even quotes someone saying that a defense of slavery was an "idiotic" explanation.

I don't read the post as making any claim about what is ultimately right or wrong. Rather, I read it as a caution similar to the common points of "how sure are you that you would have made the morally correct choice if you... (read more)

Many commenters seem to be reading this post as implying something like slavery and violence being good or at least morally okay... I read it as a caution similar to the common points of "how sure are you that you would have made the morally correct choice if you had been born as someone benefiting from slavery back when it was a thing" combined with "the values that we endorse are strongly shaped by self-interest and motivated cognition"

 

I don't agree with your characterization of the post's claims. The title is synonymous with "morality is arbitrary... (read more)

Then you quote Samuel Cartwright "conjuring up creatively compelling excuses" for slavery, and never argue against the quotation.

Do you mean this quote?

Gurwinder cites exactly such an example with the 19th century physician Samuel A. Cartwright:

A strong believer in slavery, he used his learning to avoid the clear and simple realization that slaves who tried to escape didn’t want to be slaves, and instead diagnosed them as suffering from a mental disorder he called drapetomania, which could be remedied by “whipping the devil” out of them. It’s an explanatio

... (read more)
4jimrandomh18d
Ack, I misread that, sorry. Will edit the grandparent comment to remove that part.

outweigh an extra 3 to 7 years of working on alignment

Another relevant-seeming question is the extent to which LLMs have been a requirement for alignment progress. It seems to me like LLMs have shown some earlier assumptions about alignment to be incorrect (e.g. pre-LLM discourse had lots of arguments about how AIs have to be agentic in a way that wasn't aware of the possibility of simulators; things like the Outcome Pump thought experiment feel less like they show alignment to be really hard than they did before, given that an Outcome Pump driven by somet... (read more)

2Eli Tyre18d
Here's a paraphrase of the way I take you to be framing the question. Please let me know if I'm distorting it in my translation. How's that as a summary?  

Thanks, I'd tried self-administered EMDR sometime before and didn't get much out of it. Now I gave it another shot and it caused some stuff to surface, so seemed to be doing at least something even if I didn't get to the root of the issue yet.

Do you have any thoughts on how I should try to balance the external stimuli vs. the internal content? I notice that it's easy for either the EMDR stimuli to push the emotional content out of consciousness or vice versa. Should I try to keep them exactly balanced, or predominantly emotional content with some stimuli, ... (read more)

4Anton Rodenhauser20d
These are all excellent questions! Unfortunately, I don't have definite answers. I've read somewhere that the idea is to tax the working memory as much as possible such that you can just barely hold an emotional felt sense at the same time as well. I'd be very interested if someone does some more reading and research on this! What I personally do: The more intensive the felt sense feels, the harder I focus on the EMDR "distractions", and vice-versa. 

That survey result feels hard to square with reports like this:

Three weeks ago I went to a soccer match between Shanghai SIPG and FC Seoul. After the game the traffic around the area was quite heavy. I was waiting for a pedestrian light to turn green when a couple in their electric scooter went through a red light, an old lady hit them and the three of them fell to the ground. The couple got up, yelled something to the old lady and then just got on the scooter and left. The old lady stayed there for some minutes while people passing by didn’t even try to h

... (read more)
4Richard_Ngo21d
Oh, that's very interesting. Yeah, this seems like it might account for the discrepancy here. But my instinct is that I want to hang on to the "trust" terminology, and just hold that authoritarian cultures have an impoverished definition of trust (compared with the one I gave earlier: "letting another agent do as they wish, without trying to control their behavior, because you believe that they’ll take your interests into account"). My guess is that this would be misleading in other ways—in particular, we don't typically think of freedom as a property of relationships, but rather a property of individuals.  

Oops, never got around answering this question.

When you ask how likely it is that it's an artifact of the therapeutic procedure, what's the alternative hypothesis you have in mind? What would not being an artifact of the therapeutic procedure mean?

My assumption has been that Bing was so obviously rushed and botched that it's probably less persuasive of the problems with aligning AI than ChatGPT is. To the common person, ChatGPT has the appearance of a serious product by a company trying to take safety seriously, but still frequently failing. I think that "someone trying really hard and doing badly" looks more concerning than "someone not really even trying and then failing".

I haven't actually talked to any laypeople to try to check this impression, though.

The majority of popular articles also seem t... (read more)

Is there a scenario where you could get the public concern without the hype and funding? (The hype seems to be a big part of why people are getting concerned and saying we should stop the rush and get better regulation in place, in fact.)

It seems to me that the hype and funding is inevitable once you hit a certain point in AI research; we were going to get it sooner or later, and it's better to have it sooner, when there's still more time to rein it in.

4Arthur Conmy23d
I agree that some level public awareness would not have been reached without accessible demos of SOTA models. However, I don’t agree with the argument that AI capabilities should be released to increase our ability to ‘rein it in’ (I assume you are making an argument against a capabilities ‘overhang’ which has been made on LW before). This is because text-davinci-002 (and then 3) were publicly available but not accessible to the average citizen. Safety researchers knew these models existed and were doing good work on them before ChatGPT’s release. Releasing ChatGPT results in shorter timelines and hence less time for safety researchers to do good work. To caveat this: I agree ChatGPT does help alignment research, but it doesn’t seem like researchers are doing things THAT differently based on its existence. And secondly I am aware that OAI did not realise how large the hype and investment would be from ChatGPT, but nevertheless this hype and investment is downstream of a liberal publishing culture which is something that can be blamed.

Asked what FOOM stands for, ChatGPT hallucinated a backronym of "Fast Onset of Overwhelming Mastery" among others. I like that one.

I think OpenAI has been a net-positive influence for reducing x-risk from AI, mainly by releasing products in a sufficiently helpful-yet-fallible form that society is now able to engage in less-abstract more-concrete public discourse to come to grips with AI and (soon) AI-risk.

I have a similar feeling: I think that ChatGPT has been, by far, the best thing to happen to AI x-risk discussion since the original Sequences. Suddenly a vast number of people have had their intuitions about AI shifted from "pure science fiction" to "actually a thing", and the va... (read more)

the various failure modes that ChatGPT has are a concrete demonstration both about the general difficulty of aligning AI and some of the specific issues more specifically

By this logic, wouldn't Microsoft be even more praiseworthy, because Bing Chat / Sidney was even more misaligned, and the way it was released (i.e. clearly prioritizing profit and bragging rights above safety) made AI x-risk even more obvious to people?

I agree that ChatGPT was positive for AI-risk awareness. However from my perspective being very happy about OpenAI's impact on x-risk does not follow from this. Releasing powerful AI models does have a counterfactual effect on the awareness of risks, but also a lot of counterfactual hype and funding (such as the vast current VC investment in AI) which is mostly pointed at general capabilities rather than safety, which from my perspective is net negative.

Thanks for sharing this! Because of strong memetic selection pressures, I was worried I might be literally the only person posting on this platform with that opinion.

This essay has some tips on that, starting from the "More patterns of Anki use". There are also various LW articles about Anki under the Spaced Repetition tag, some of them such as My Anki Patterns have card design tips.

Hmm, two individuals of a species mating obviously couldn't compare their genomes with other representatives of the species and take the modal allele. But many species, especially plants, do carry more than two copies of each chromosome (e.g. black mulberry apparently has 44 copies of each gene). How difficult would it be to evolve a process that compared the alleles on each chromosome that the individual carried and picked the modal one for producing gametes?

Intuitively it feels to me like it'd be hard for biology to do/evolve and that it'd require someth... (read more)

4cousin_it25d
Yeah, it would have to be at least 3 individuals mating. And there would be some weird dynamics: the individual that feels less fit than the partners would have a weaker incentive to mate, because its genes would be less likely to continue. Then the other partners would have to offer some bribe, maybe take on more parental investment. Then maybe some individuals would pretend to be less fit, to receive the bribe. It's tricky to think about, maybe it's already researched somewhere?
7TekhneMakre25d
Naively extrapolating, such an allele couldn't spread, because before it's very common, it would delete itself!

The Stanley Parable is a video game originally released in 2013

It was also released before that as a Half-Life 2 mod in 2011.

I played the 2013 version; I didn't find it to be that interesting in terms of conventional philosophical questions, but I did find it to be pretty hilarious and insightful when it came to poking fun at narrative and game-design conventions. Recommended.

Cool, I read a bit of the rulebook and it seemed neat!

Possibly worth noting: I linked this post in a local SSC/ACX chat and one of the reactions that I got was:

I got a mild skeptical distaste reaction because they put the "this is useful for AI safety redteaming" aspect on the forefront, instead of *fun*.

And I have to admit that I did also have a bit of the same reaction when I first read the post; the phrasing was the kind that I associate with awful political art, giving me a prior expectation of "this probably isn't very good". Reading the beginning of ... (read more)

6Daniel Kokotajlo25d
Thanks! Duly noted, thanks for the feedback. I agree that political art is typically awful. FWIW, The Treacherous Turn was approximately 100% optimized to be fun. We all knew from the beginning that it wouldn't be useful if it wasn't fun. We did put some thought into making it realistic, but the realism IMO adds to the fun rather than subtracts; I would still have included it even if I didn't care about impact at all.  

Well this was an interesting experience to re-read (and I had apparently totally forgotten about having read this once before, since I was surprised by my comment from two years back in the comment section, after having missed the note in the beginning of this being republished)

Notes from while reading this:

I began to do and feel something I call “trying to squirm out from under the problem”.

Oh, this feels familiar. Emphasis on feels. There's a reaction I've sometimes noticed my mind doing, where it feels like I'm moving towards something unpleasant like a... (read more)

4Kaj_Sotala1mo
(I did the exercise)

Excitement without a solid foundation of caution is what is causing the current AI race.

To some extent - though Google's statements about having to join the race and forget about their earlier more cautious policy now that OpenAI/Microsoft decided to rush ahead, sound fear-based.

Updated the post with excerpts from the MIT Technology Review video interview, where Hinton among other things brings up convergent instrumental goals ("And if you give something the ability to create its own sub-goals in order to achieve other goals, I think it'll very quickly realise that getting more control is a very good sub-goal because it helps you achieve other goals. And if these things get carried away with getting more control, we're in trouble") and explicitly says x-risk from AI may be close ("So I think if you take the existential risk seriou... (read more)

I edited the title and introductory paragraph to read "How I apply (so-called) Non-Violent Communication" to help signal that I don't endorse the implication.

So there's something to that, but I'm a little wary about taking that interpretation too far. Taken far enough, it implies that if group A has a sensible take on a concept, then as soon as a group B shows up that has a bad take on it, you can use it to discredit A as a motte for B. It seems bad if we can discredit any concept - including valuable ones - just by making up a bad take on it and spreading it.

I talked about that in this post:

But suppose that we were discussing something of which there were both sensible and crazy interpretations - held by diffe

... (read more)

It's a motte and bailey: the people who use the word as part of a technical term clearly and explicitly disavow the implication, but other people clearly and explicitly call out the implication as if it were fact.

If some people consistently and explicitly disavow the implication, but other people consistently and explicitly endorse the implication, then I don't think that that's motte and bailey? As I understand it, M&B involves the same person being inconsistent about the meaning, not different people sticking to consistent but conflicting interpretations; that's just people disagreeing with each other.

6cwillu1mo
My understanding is that M&B is intended to be broader than that, as per: “So it is, perhaps, noting the common deployment of such rhetorical trickeries that has led many people using the concept to speak of it in terms of a Motte and Bailey fallacy. Nevertheless, I think it is clearly worth distinguishing the Motte and Bailey Doctrine from a particular fallacious exploitation of it. For example, in some discussions using this concept for analysis a defence has been offered that since different people advance the Motte and the Bailey it is unfair to accuse them of a Motte and Bailey fallacy, or of Motte and Baileying. That would be true if the concept was a concept of a fallacy, because a single argument needs to be before us for such a criticism to be made. Different things said by different people are not fairly described as constituting a fallacy. However, when we get clear that we are speaking of a doctrine, different people who declare their adherence to that doctrine can be criticised in this way. Hence we need to distinguish the doctrine from fallacies exploiting it to expose the strategy of true believers advancing the Bailey under the cover provided by others who defend the Motte.” [bold mine] http://blog.practicalethics.ox.ac.uk/2014/09/motte-and-bailey-doctrines/ [http://blog.practicalethics.ox.ac.uk/2014/09/motte-and-bailey-doctrines/]

An abuser has an emotional need for respect. He experiences it as deeply hurtful when his partner has conversations with other men. When she talks to other men anyway, he feels betrayed. He says “When you talk to other men, I feel hurt because I need mutual respect.”

Using NVC principles, how do you say that what he is doing is wrong?

NVC generally wouldn't say that having a need is wrong by itself. Rather its defense against unreasonable demands is to emphasize that when responding to a request from someone else, you should first check how those fit with yo... (read more)

Do you mean that saying "my method of communication is non-violent communication" implies that everyone else is communicating violently? That's a reasonable point; I hadn't really thought about it, since I'd been mostly treating NVC as a technical term or proper noun rather than as something that was intended to communicate literal meaning. (Especially since it's often referred to as just "NVC", so you don't necessarily even say the words.)

To be clear, I don't mean to imply that, and I don't subscribe to the interpretation that people who don't use NVC are... (read more)

9jimmy1mo
  I'll bite that bullet. People who aren't communicating in the spirit of NVC are "communicating violently". Not in the sense of "Words are literal violence!" because "sticks and stones", but in the sense that "If you don't give me what I want I will use sticks and stones to break your bones" is "communicating violently". NVC points at the important insight that much of what passes for "normal communication" is actually subtle and implicit threats, which can and do escalate to real physical harm and literal violence. "Why are you being so mean?" doesn't pass for NVC, and that's not unrelated to the fact that it can be used to recruit someone to do violence on your behalf against the person who you accuse of doing you wrong. It doesn't usually get that far, in the same way that parking tickets aren't usually enforced with guns drawn, but there's a reason that libertarians like to point out that all laws are ultimately enforced at gunpoint and the same thing applies here. That doesn't mean that we should "must" at people who aren't communicating in the spirit of NVC, because as you point out, that would be violating the spirit of NVC. But I do think the term fits, and the way to get around the hubris of saying "my method of communication is nonviolent communication!" is to 1) point out how the term "violence" is actually legit and doesn't just mean "offensive", 2) don't run around claiming that you actually succeed at doing it more than you do, and 3) point out how "nonviolent" isn't even the goal to aspire to 100% of the time and definitely not synonymous with "good".
2localdeity1mo
That kind of thing, yes.  I should mention that I have no systematic perspective here—I've had several acquaintances mention they've learned about NVC, and seen various internet discussions, but I have no idea what the "usual" or "average" usage is like.  (I'll also mention that I think at least some of the central techniques are good ones—I've elsewhere encountered the formulation of "I statements", as in "I see X" and "When Y, I feel Z".) I have seen a few people say that abusive people have used NVC as a tool.  Essentially using it as a way of communicating, legitimizing, and lending weight to their unreasonable desires.  Googling, I found this [https://realsocialskills.org/2014/07/17/nonviolent-communication-can-be-emotionally-violent/] (I don't endorse everything this article says, and much of it is "I have no idea how often what you're saying happens in practice", but posed as a hypothetical it makes sense): My memory traces also include (a) someone saying he and his ~10-year-old kid took a class on it, and (b) a boss using it with her employees [although the source I found on this seems to be a hypothetical], both cases leading to discovering how to use it for emotional blackmail.  (I think someone opined that NVC shouldn't be used when there is a power disparity; I wonder if this is common advice.) I mean, to some extent any improved communication technique is going to be a tool that increases the options available to an abuser, especially when their interlocutor isn't very sophisticated.  The existence of misusers doesn't prove it's bad on net. Still, having the name be as virtuous-sounding as "Nonviolent Communication" seems to make a couple of things easier: 1. pressuring someone to go along with it ("How could you possibly object to this?", or someone anticipating that response and staying silent) 2. practitioners and teachers being blithely unaware of possible misuses by themselves or others It may also turn away some of the more scrupul
9cwillu1mo
I'm deeply suspicious of any use of the term “violence” in interpersonal contexts that do not involve actual risk-of-blood violence, having witnessed how the game of telephone interacts with such use, and having been close enough to be singed a couple times. It's a motte and bailey: the people who use the word as part of a technical term clearly and explicitly disavow the implication, but other people clearly and explicitly call out the implication as if it were fact. Accusations of gaslighting sometimes follow. It's as if “don't-kill-everyoneism” somehow got associated with the ethics-and-unemployment branch of alignment, but then people started making arguments that opposing, say, RLHF-imposed guardrails for proper attribution, implied that you were actively helping bring about the robot apocalypse, merely because the technical term happens to include “kill everyone”. Downside of most any information being available to use from any context, I guess.

To get away from the analogies, I really appreciate this piece and how it was written. I specifically appreciate it because it doesn't feel like it is an attempt to make me more vulnerable to something bad. Also I think it might have helped me get a bit of a felt sense shift.

Thank you for sharing that, I'm happy to hear it. :)

As the other comment pointed out, I'm not assuming that one could control their emotions - I actually lean towards thinking that attempts to control one's emotions are often harmful, though of course there's also a place for healthy emotion regulation.

To be specific, I’m pointing to language like « should feel », « rational to feel » etc.

This clarification seems relevant. 

Also in general, I don't think that considering some feelings more rational than others requires an ability to control one's feelings. A feeling can be instrument... (read more)

1Aorou1mo
Thanks for pointing to your clarification. I find it a lot clearer than the OP.

Right, that makes sense.

And to clarify, as I tried to say in the introduction, the post is mostly intended to counter the thought that "I shouldn't feel safe". So if someone is having thoughts that it's wrong to feel safe and they should stop doing so, then the intent of the post isn't to say "here's how you should feel". Rather, it's just to say "if you do feel safe, I don't think you need to take a metaphorical hammer and hit yourself with it until you feel unsafe (nor do you need to believe people who say that you should); here's why I think you can sto... (read more)

4Dagon1mo
Yup, thanks for the clarification.

Agree. (I'm not saying that losing one's trust in civilizational adequacy is necessarily a bad thing on net, just that it can also lead to some maladaptive thought patterns.)

Good question, I think often there's been a failure to differentiate going on. Though it's been quite a while since I spoke to some of the people I was thinking of so my recollection of them might be misleading (and others I've only heard about through second-hand accounts).

Is your model that our thoughts come first, and feelings second?

I think that there are cases where that's true, but that generally our emotional state exerts a strong influence on what kinds of thoughts we're capable of having. So feeling safe (or at least not feeling unsafe) may be a prerequisite for being able to think clearly about risks.

(Though this gets complicated because there are influences going in both directions - if I thought that intellectual ideas had zero influence on feelings, it would have been pointless for me to write this post.)

4Dagon1mo
Not exactly - there's more feedback loop than that.  I fully agree with "this gets complicated". I would say that intentional changes to mind-state tend to be thoughts-first.  I don't know if that's tautological from the nature of "intentional", but it does seem common enough to make it the best starting point for most people.

I don't have a driving license so this isn't a situation I'd have personal experience with, but I imagine that it would be useful to have some degree of unsafeness to focus your attention more strongly on the driving.

It's the kind of thought that one might have if they have a (possibly low-grade) anxiety issue: you feel anxious and like the world isn't safe and you need to be alert all the time, so then your mind takes that observation as an axiom and generates intellectual reasoning to justify it. And I think there's a subset of rationalists who were driven to rationality because they were anxious; Eliezer even has an old post suggesting that in order to be really dedicated to rationality, you need to have undergone trauma that broke your basic trust in people:

Of the

... (read more)
3MondSemmel1mo
This logic can be taken too far - I don't see the point of feeling constantly anxious -, but at least on an intellectual level, I think it does make a certain amount of sense. It's hard to notice the insanity or inadequacy of the world until it affects you personally. Some examples of this: * People buy insurance to be safe from <disaster>, but insurance companies often don't want to pay out. So when you buy insurance, you might incorrectly feel safe, but only notice that you weren't if a disaster actually happens. * If you've never been ill, then it's easy to believe that if you got ill, you could just go to the doctor and be healed. Sometimes things do work that way. At other times, you might learn that reality is more complicated, and civilization less competent, than previously thought. * I think the Covid pandemic, and the (worldwide!) inadequate policy response, should've been at least a bit traumatizing to every person on this planet. Not necessarily on an emotional level, but certainly on an intellectual level. There's a kind of trust one can only have in institutions one knows ~nothing about (related: Gell-Mann amnesia), and the pandemic is the kind of event that should've deservedly broken this kind of trust.
7havequick1mo
Re "they should also pop out in a sane person's mind within like 15 seconds of actually hearing that statement" I agree with that in the abstract; few people will say that a state of high physiological alertness/vigilance is Actually A Good Idea to cultivate for threats/risks not usefully countered by the effects of high physiological alertness. Being able to reason about that in the abstract doesn't necessarily transfer to actually stopping doing that. Like personally, I feel like being told something along the line of "you're working yourself up into a counterproductive state of high physiological alertness about the risks of [risk] and counterproductively countering that with incredibly abstract thought disconnected from useful action" is not something I am very good at hearing from most people when I am in that sort of extraordinarily afraid state. It can really feel like someone wants to manipulate me into thinking that [risk] is not a big deal, or discourage me from doing anything about [risk], or that they're seeking to make me more vulnerable to [risk]. These days this is rarely the case; but the heuristic still sticks around. Maybe I should find its commanding officer so it can be told by someone it trusts that it's okay to stand down... With the military analogy; it's like you'd been asked to keep an eye out for a potential threat, and your commanding officer tells you on the radio to get on REDCON 1. Later on you hear an unfamiliar voice on the radio which doesn't authenticate itself, and it keeps telling you that your heightened alertness is actually counterproductive and that you should stand down. Would you stand down? No, you'd be incredibly suspicious! Interfering with the enemy's communication is carte blanche [https://en.wikipedia.org/wiki/Ruse_de_guerre#Legitimate_ruses] in war. Are there situations where you would indeed obey the order from the unfamiliar voice? Perhaps! Maybe your commanding officer's vehicle got destroyed, or more prosaicall

I thought I saw some in Reddit discussion but couldn't quickly find those comments anymore, also at least one of my Facebook friends.

Another interview with Hinton about this: https://www.technologyreview.com/2023/05/02/1072528/geoffrey-hinton-google-why-scared-ai/

Chosen excerpts:

People are also divided on whether the consequences of this new form of intelligence, if it exists, would be beneficial or apocalyptic. “Whether you think superintelligence is going to be good or bad depends very much on whether you’re an optimist or a pessimist,” he says. “If you ask people to estimate the risks of bad things happening, like what’s the chance of someone in your family getting really sick or bei

... (read more)

I watched it spend tens of trillions of FLOPs to write out, in English, how to do a 3x3 matrix multiplication. It was so colossally inefficient, like building a humanoid robot and teaching it to use an abacus.

There's also the case where it's allowed to call other services that are more optimized for the specific use case in question, such as querying Wolfram Alpha:

The post otherwise makes sense to me, but I'm confused by this bit:

It can do better if it's allowed to run algorithms by "thinking out loud". It's really slow, and this is a good way to fill up its context buffer. The slowness is a real problem - if it outputs ~10 token/sec, it will take forever to solve any problems that are actually both big and hard. This is a neat trick, but it doesn't seem like an important improvement to its capabilities.

Why not?

It seems like humans also run into the same problem - the brain can only do a limited amount of inference ... (read more)

4Jonathan Marcus2mo
This was actually my position when I started writing this post. My instincts told me that "thinking out loud" was a big enhancement to its capabilities. But then I started thinking about what I saw. I watched it spend tens of trillions of FLOPs to write out, in English, how to do a 3x3 matrix multiplication. It was so colossally inefficient, like building a humanoid robot and teaching it to use an abacus. Then again, your analogy to humans is valid. We do a huge amount of processing internally, and then have this incredibly inefficient communication mechanism called writing, which we then use to solve very hard problems!  So my instincts point both ways on this, but I have nothing resembling rigorous proof one way or the other. So I'm pretty undecided.
Load More