This is a special post for quick takes by FinalFormal2. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Doesn't this basically sum up the course of the rationalist project?
I know we're more about AI doom at this point, but if anyone is still interested in raising the sanity waterline, it seems like it's important to make it common knowledge that teaching about biases isn't enough to make people sane.
From my spot outside of the loop, it seems like that's about where the project stopped.
What's the cutting edge on raising the sanity waterline? Are people still working on this project? What might I have missed?
Seems like the empiric lesson is that it is dangerous to be a half-rationalist, and we do not have a good way to test who would be helped and who would be hurt by learning about rationality.
There are people who benefited from being introduced to LW-style rationality and the rationalist community in general, sometimes a lot. (I include myself in this group.)
There are also people who got hurt, or hurt others, as a result of being exposed to the rationalist memes and the community. (A long list culminating with Zizians.)
We communicate openly online, so there are no gatekeepers to the knowledge; no driving license.
Introduce some secrecy? That seems to increase harm: some problematic groups (Zizians, Leverage Research) positioned themselves aside from the mainstream rationalist community, and didn't communicate their original insights. The result was a dramatic decrease of sanity, as even obviously wrong and harmful ideas could propagate in a small group supported by the charisma of its leader.
Introduce more common sense and conservatism? Seems like a post-rationalist project, which from my perspective is just watered-down rationalism. (Plus Buddhism, which is like... how the fuck can people who have read the Sequences and understood how mysterious answers are wrong and religions are silly, suddenly embrace some stupid thousand years old religion, just because it has promised them superpowers if they meditate hard enough? After a few years of experimenting, does anyone already have the superpowers, or is it that you just have to keep believing and keep practicing indefinitely without expecting any experimentally verifiable results?) I frankly, do not see any higher sanity here. It is just the normie epistemology, with all of its advantages and disadvantages.
Perhaps we should have a group of "certified sane rationalists", and a way to ask them, if you have a crazy mindblowing idea. (Creating the group would be relatively simple: start with some core, e.g. Eliezer Yudkowsky and Anna Salamon and Scott Alexander, and then add or remove people based on majority vote of the current group members.) The problem is, either most people wouldn't bother asking them, or they would be too busy responding.
There are people who benefited from being introduced to LW-style rationality and the rationalist community in general, sometimes a lot. (I include myself in this group.)
Could you expand more on this? Not so much on the community side, but on the idea of rationality? What's your experience of improvement there?
I don't think I really believe that you experienced serious benefits from learning rationality.
I wish I had better information about this, I'm mostly just going off of vibes. I also wish I had better information about the state of the art of sanity-waterline raising.
I think there's also two ways to break down the sanity project, which is one- can you make a receptive audience rational or improve their lives through rationality, and two, can you raise the societal sanity waterline, like make the average person saner.
I'm interested in the model where schools are generally a big cause of the problem of societal irrationality, but that doesn't seem like something people talk about much in this community.
Well, we don't get to see the parallel timeline, so it is difficult to say precisely what is the difference. Intuitively, I compare the previous state to the current state, but that assumes that in the parallel timeline "nothing happened". Perhaps other interesting things have happened there.
The most obvious change in my behavior immediately after reading the Sequences: I stopped debating politics online. (Previously I wasted a lot of time doing that. Although now I am spending that much time on LW and ACX.) Suddenly, debating politics online felt like talking to retards: people were making the same obvious mistakes over and over again with no intention to ever learn. I think this was good for my mental health.
I have also met a few friends in the rationalist community. As a result of our discussions I started to care about my health more, and bought some bitcoins. I think some of these things would not happen in most of the parallel timelines. I probably use AIs a bit more and better than I would without reading LW.
I wish I had some more impressive results, but I am happy even for these improvements. (My excuse is that I have small kids and the community in this part of the world is very small. My benefits seem to come mostly as a result of interacting with other rationalists. I guess it is much easier for me to take ideas seriously when I also receive some social support for that. Reading alone does not have the same effect.)
I know some people who met each other on LW meetups, started doing some crypto business together, now they are rich and... moved away and I lost regular contact with them, sadly. I am not saying here that crypto = rationality. But it was the rationalist community that allowed them to meet each other: smart people who share some perspective and can trust each other's sanity.
Across the world, I think Scott Alexander has benefited a lot from the community. It would make more sense to ask him what specifically he attributes to it.
I think there's also two ways to break down the sanity project, which is one- can you make a receptive audience rational or improve their lives through rationality, and two, can you raise the societal sanity waterline, like make the average person saner.
In my case (not sure how typical), the greatest value seems to come from small local groups. Local, because meeting people in real life seems better than chatting with them online; our monkey brains treat we meet as "more real" than the ones who only seem to exist on the screen. (That's probably a good intuition for the era of online bots.)
There is a lot of value in taking five minutes by the clock to actually think about a problem. I think there is even more value in taking fifteen minutes to talk about a problem with your trusted fellow rationalist friends. There is something powerful in having people whose opinion you can trust, who practice some basic epistemic hygiene so that their advice does not contain things like "you have to pray" or "that's fate" or "hey, try this scam I found online, it will totally work".
I'm interested in the model where schools are generally a big cause of the problem of societal irrationality, but that doesn't seem like something people talk about much in this community.
There are many articles about education, many of them critical. (It's just difficult to find them among all those AI-related posts.) I think the consensus is that schools are mostly a waste of time, a very costly signal of conformity that many employers want.
(I don't think this is a complete waste. There are people who are unable to keep a job because they are simply unable to wake up and come to the job every day consistently. The society benefits from a mechanism that trains them and certifies this. It's just a huge waste of time for everyone smarter and more disciplined than that. Without the school system the society would probably split to a small group of homeschooled geniuses, a medium-sized of kids with mixed results approximately as good as they have now, and the largest group of completely unemployable idiots at the bottom. So far, we still the people at the bottom to be able to get jobs. Also, the idea of even stupider voters in democracy is scary.)
The perspective of school not merely as a waste of time but actively harmful... well, there is the "teacher's password" anti-pattern, but I do not remember anything more in this direction.
What do we have for hypotheses about how people acquire false beliefs at scale?
Michael Vassar drew my attention to this problem- and coming from a Christian background it fascinates me how well "religion is a conspiracy" maps onto my lived experience.
Motivated reasoning and Echo chambers.
Our decision-making mechanisms are guided by predictions of reward. Being right predicts reward, but so does giving answers your peers will like (we're wired for social reward, plus they'll give you food and shelter if they like you). So those two signals are mixed together when we make decisions about what evidence to look at, what lines of thought to follow, and ultimately what we believe. Thus, our reasoning is biased toward answers and beliefs that are in our (perceived) best interests.
Echo chambers are pretty obvious and have existed long before social media; the evidence/arguments one tends to hear is highly correlated with their social environment, and we self-select our social environments. We stay around those who agree with us and we agree with those around us.
Echo chambers are pretty obvious and have existed long before social media
Yeah, internet gives you a wide choice of echo chambers, so you are not limited to the ones you were born in or the ones you could find around you.
Even better, you can visit multiple echo chamber pseudonymously, which is something that would be very difficult to do in real life, as many echo chambers would punish that.
Lots of beliefs held at scale we would think of as false: e.g., the earth being the centre of the universe. I think the reason is we rely on justification to shore up our beliefs and other people believing things is justification. https://www.lesswrong.com/posts/hcymnEAKtwvED7Y8o/what-are-we-actually-evaluating-when-we-say-a-belief-tracks
I believe that we're going to see heavy political and social instability over the next 5 years, how do I maximize my EV in light of this? Primarily I'm thinking about financial investments.
Some things I was thinking about: Gold in GDX, Cybersecurity in HACK, Options income in JEPI, Defense/Aerospace in ITA
What's the deal with AI welfare? How are we supposed to determine if AIs are conscious and if they are, what stated preference corresponds to what conscious experience?
Surely the AIs can be trained to say "I want hugs" or "I don't want hugs," just as easily, no?
We haven't figured it out for humans, and only VERY recently in history has the idea become common that people not kin to you deserve empathy and care. Even so, it's based on vibes and consensus, not metrics or proof. I expect it'll take less than a few decades to start recognizing some person-hood for some AIs.
It'll be interesting to see if the reverse occurs: the AIs that end up making decisions about humans could have some amount of empathy for us, or they may just not care.
There are a lot of good reasons to believe that stated human preferences correspond to real human preferences. There are no good reasons that I know of to believe that any stated AI preference corresponds to any real AI preference.
"Surely the AIs can be trained to say "I want hugs" or "I don't want hugs," just as easily, no?"
There are a lot of good reasons to believe that stated human preferences correspond to real human preferences.
Can you name a few? I know of one: I assume that there's some similarity with me in because of similar organic structures doing the preferring. That IS a good reason, but it's not universally compelling or unassailable.
Actually, can you define 'real preferences' in some way that could be falsifiable for humans and observable for AIs?
"Surely the AIs can be trained to say "I want hugs" or "I don't want hugs," just as easily, no?"
Just as easily as humans, I'm sure.
"Surely the AIs can be trained to say "I want hugs" or "I don't want hugs," just as easily, no?"
Just as easily as humans, I'm sure.
No. The baby cries, the baby gets milk, the baby does not die. This is correspondence to reality.
Babies that are not hugged as often, die more often.
However, with AIs, the same process that produces the pattern "I want hugs" just as easily produces the pattern "I don't want hugs."
Let's say that I make an AI that always says it is in pain. I make it like we make any LLM, but all the data it's trained on is about being in pain. Do you think the AI is in pain?
What do you think distinguishes pAIn from any other AI?