This is a special post for quick takes by Jonathan Claybrough. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Jonathan Claybrough's Shortform
116Jonathan Claybrough
34Jsevillamol
23Neel Nanda
17Siebe
10Satron
10Chris_Leong
19Jonathan Claybrough
11Valentine
5Said Achmiz
1Valentine
7Said Achmiz
4Valentine
8Said Achmiz
4Valentine
14Ann
2Said Achmiz
4Zack_M_Davis
1Valentine
4Said Achmiz
2Valentine
2Said Achmiz
2Chris_Leong
9Valentine
4Chris_Leong
4Valentine
11Richard_Kennaway
6Valentine
4Richard_Kennaway
7Rafael Harth
6Tensor White
4dsatan
12Jonathan Claybrough
6Jonathan Claybrough
4Jonathan Claybrough
4Jonathan Claybrough
EpochAI seem do be doing a lot of work that'll accelerate AI capabalities research and development (eg. informing investors and policy makers that yes AI is a huge economic deal and here are the bottlenecks you should work around, building capabilities benchmarks to optimize for). Under common-around-LW assumptions that no one could align AGI at this point, they are, by these means, increasing AI catastrophic and existential risk.
At a glance they also seem to not be doing AI x-risk reducing moves, like using their platform to mention that there are risks associated to AI, that these are not improbable, and that these require both technical work and governance to manage appropriately. This was salient to me in their latest podcast episode - speaking at length about AI replacing human workers in 5 to 10 years and the impact on the economy, without even hinting that there are risks associated with this, is burying the lede.
Given that Epoch AI is funded by OpenPhilantropy and Jaan Tallinn, who on their face care about reducing AI x-risk, what am I missing? (non rhetorical)
- What is EpochAi's theory of change for making the world better on net?
- Overall, is EpochAI increasing or reducing ai x-risk (on LW models, in their models)?
I wanted this to be but a short shortpost, but since I'm questioning a pretty big maybe influential org under my true name let me show good faith with proposing reasons that might contribute to what I'm seeing. For anyone unaware of their work, maybe check out their launch post, or their recent podcast episode.
- OpenPhil is unsure about magnitude of AI x-risk so invest in forecasting AI capabilities to know if they should invest more in AI safety.
- EpochAI doesn't believe AI x-risk is likely and believes that accelerating is overall better for humanity (seems true for some employees but not all)
- EpochAI believes that promoting the information that AGI is economically important and possible soon is better because governments will better govern it than counterfactually
- EpochAI is saying what they think is true without selection to avoid being deceptive (this doesn't mesh with the next reason)
- EpochAI believe that mentionning AI risks at this stage would hurt their platform and their influence, and are waiting for a more ripe opportunity/better argued paper.
Tagging a few EpochAI folk that appeared in their podcast - @Jsevillamol @Tamay @Ege Erdil
I'm talking from a personal perspective here as Epoch director.
- I personally take AI risks seriously, and I think they are worth investigating and preparing for.
- I co-started Epoch AI to get evidence and clarity on AI and its risks and this is still a large motivation for me.
- I have drifted towards a more skeptical position on risk in the last two years. This is due to a combination of seeing the societal reaction to AI, me participating in several risk evaluation processes, and AI unfolding more gradually than I expected 10 years ago.
- Currently I am more worried about concentration in AI development and how unimproved humans will retain wealth over the very long term than I am about a violent AI takeover.
- I also selfishly care about AI development happening fast enough that my parents, friends and myself could benefit from it, and I am willing to accept a certain but not unbounded amount of risk from speeding up development. Id currently be in favour of slightly faster development, specially if it could happen in a less distributed way. I feel very nervous about this however, as I see my beliefs as brittle.
I'm also going to risk also sharing more internal stuff without coordinating on it, erring on the side of over sharing. There is a chance that other management at Epoch won't endorse these takes.
- At management level, we are choosing to not talk about risks or work on risk measurement publicly. If I try to verbalize it, it's due to a combination of us having different beliefs on AI risk, which makes communicating from a consensus view difficult, believing that talking about risk would alienate us from stakeholders skeptical of AI Risk, and the evidence about risk being below what we are comfortable writing about.
- My sense is that OP is funding us primarily to gather evidence relevant to their personal models. Eg two senior people at OP particularly praised our algorithmic progress paper because it directly informs their models. They do also care about us producing legible evidence on key topics for policy, such as the software singularity or post training enhancements. We have had complete editorial control and I feel confident in rejecting topic suggestions that come from OP staff when they don't match my vision of what we should be writing about (and have done so in the past).
- In term of overall beliefs, we have a mixture of people very worried and skeptical of risk. I think the more charismatic and outspoken people at Epoch err towards being more skeptical of risks, but no one at Epoch is dismissive of it.
- Some stakeholders I've talked to have expressed this view that they wish for Epoch AI to gain influence and then communicate publicly about AI risk. I don't feel comfortable with that strategy, one should expect Epoch AI to keep a similar level of communication about risk as we gain influence. We might be willing to talk more about risks if we gather more evidence of risk, or if we build more sophisticated tools to talk about it, but this isn't the niche we are filling or that you should expect us to fill.
- The podcast is actually a good example here. We talk toward the end about the share of the economy owned by biological humans becoming smaller over time, which is an abstraction we have studied and have moderate confidence in. This is compatible with an AI takeover scenario, but also a peaceful transition to an AI dominated society. This is the kind of communication about risks you can expect from Epoch, relying more on abstractions we have studied than stories we don't have confidence in.
- The overall theory of change of Epoch AI is that having reliable evidence on AI will help raise the standards of conversation and decision making elsewhere. To be maximally clear, we are willing to make some tradeoffs like publishing work like FrontierMath and our distributed training paper that plausibly speed up AI development in service of that mission.
This seems fine to me (you can see some reasons I like Epoch here). My understanding is that most Epoch staff are concerned about AI Risk, though tend to longer timelines and maybe lower p(doom) than many in the community, and they aren't exactly trying to keep this secret.
Your argument rests on an implicit premise that Epoch talking about "AI is risky" in their podcast is important, eg because it'd change the mind of some listeners. This seems fairly unlikely to me - it seems like a very inside baseball podcast, mostly listened to by people already aware of AI risk arguments, and likely that Epoch is somewhat part of the AI risk-concerned community. And, generally, I don't think that all media produced by AI risk concerned people needs to mention that AI risk is a big deal - that just seems annoying and preachy. I see Epoch's impact story as informing people of where AI is likely to go and what's likely to happen, and this works fine even if they don't explicitly discuss AI risk
1I don't think that all media produced by AI risk concerned people needs to mention that AI risk is a big deal - that just seems annoying and preachy. I see Epoch's impact story as informing people of where AI is likely to go and what's likely to happen, and this works fine even if they don't explicitly discuss AI risk
I don't think that every podcast episode should mention AI risk, but it would be pretty weird in my eyes to never mention it. Listeners would understandably infer that "these well-informed people apparently don't really worry much, maybe I shouldn't worry much either". I think rationalists easily underestimate how much other people's beliefs depend on what the people around them & their authority figures believe.
I think they have a strong platform to discuss risks occasionally. It also simply feels part of "where AI is likely to go and what's likely to happen".
Listeners would understandably infer that "these well-informed people apparently don't really worry much, maybe I shouldn't worry much either".
I think this is countered to a great extent by all the well-informed people who worry a lot about AI risk. I think the "well-informed people apparently disagree on this topic, I better look into it myself" environment promotes inquiry and is generally good for truth-seeking.
More generally, I agree with @Neel Nanda, it seems somewhat doubtful that people listening to a very niche Epoch Podcast aren't aware of all the smart people worried about AI risk.
Are people losing ability to use and communicate in previous ontologies after getting Insight from meditation ? (Or maybe they never had the understanding I'm expecting of them ?) Should I be worried myself, in my practice of meditation ?
Today I reread Kensho by @Valentine, which presents Looking, and the ensuing conversation in the comments between @Said Achmiz and @dsatan, where Said asks for concrete benefits we can observe and mostly fails to get them. I also noticed interesting comments by @Ruby who in contrast was still be able to communicate in the more typical LW ontology, but hadn't meditated to the point of Enlightenment. Is Enlightenment bad? Different ?
My impression is that people don't become drastically better (at epistemology, rationality, social interaction, actually achieving your goals and values robustly) very fast through meditating or getting Enlightened, though they may acquire useful skills that could help to get better. If that's the case, it's safe for me to continue practicing meditation, getting into Jhanas, Insight etc (I'm following The Mind Illuminated), as the failings of Valentine/dsatan to communicate their points could just be attributed to them not being able to before either.
But I remain wary that people spend so much time engaging and believing in the models and practices taught in meditation material that they actually change their minds for the worse in certain respects. It looks like meditation ontologies/practices are Out to Get You and I don't want to get Got.
I don't know how to answer the general query. But I can say something maybe helpful about that Kenshō post and "Looking":
The insight was too new. I wrote the post just 4 months after the insight. I think I could answer questions like this way, way more clearly today.
(…although my experience with Said in particular has always been very challenging. I don't know that I could help him any better today than I could in 2018. Maybe? He seems to use a mind type that I've never found a bridge for.)
The issue is that the skill needed to convey an insight or skill is often significantly different from the insight/skill itself. And in many cases, getting quite good at something can make you worse at teaching it because you can come to forget what it's like to be a beginner.
IME it's necessary to reconstruct (literally "re-member") various beginners' minds after gaining expertise. Otherwise you can't meet beginners where they are. This is the essence of PCK.
I hadn't done basically any of that when I wrote Kenshō.
So I don't think the issue is that meditation or whatever borked my ability to talk clearly. It's more that I had a massive shift and then tried to talk about it right away. This challenge is a pretty general one IMO.
I suspect the issue is that the kind of massive insight that can happen from meditation, psychedelics, & related experiences can result in pretty huge shifts, making this "I'm talking gibberish for a while" thing way more common and enduring than with (say) mathematical insights.
A simple example with Kenshō:
One benefit of learning to Look (though I don't care for this framing anymore) is getting better at attending to what matters to you, instead of attending to what you think matters to you, when there's a difference.
Naturally many mind designs will think there's no difference, or that if there is one it's pretty small. But it's often — I daresay usually — massively larger than we tend to notice. The shift can feel a bit like suddenly remembering that you have a beloved child, and realizing you've been neglecting them and even forgetting they exist because you were distracted. It's not pleasant to realize! It might make you "less functional" according to your earlier standards. But you absolutely would not want to reverse the revelation!
One challenge I was grappling with in 2018 is that many minds will insist that (a) there's no reason to trust that such an insight is valid as opposed to being a delusion and (b) even if it were valid, that maybe it's better not to have the revelation in the first place since it borks metrics you can tell are useful.
In 2018 I didn't know how to orient to that while holding my clarity. All I could do is flail my arms and say "Epistemic puzzle!" Very, very poor PCK.
I wouldn't say my PCK is pristine now. But I could have a much more clear and kind conversation about it today than I could five years ago. It just took a while to develop.
I don't think so. Not in terms that would satisfy you, best as I can tell.
Although… I wonder if we can translate a bit. If you were trying to convey this whole "falling in love" thing to me, while I'm suspicious about whether it exists or that if it exists that it has any value, and I were pressing you for "cake" about "falling in love", what would you offer?
I mean that sincerely. Those two feel like similar type errors to me. If you can offer a few examples of "cake" for falling in love then I might be able to figure out how to offer you "cake" for kenshō.
(I'm not too particular about "falling in love" per se. It's just the most fitting example that popped into mind.)
Well, to start with, the question of whether “falling in love” exists and whether it’s got any value are very different questions.
If you were unconvinced about there being any such thing as “falling in love” (having, let us suppose, never experienced it yourself, and perhaps even being aromantic), I would start by pointing you to various descriptions of the experience, from a wide variety of different cultures, eras, etc. I would then point out that these descriptions come in two types: first, phenomenal accounts, “from the inside”, of what it feels like to fall in love; and second, observational accounts, of what behavioral changes may be observed in people who have supposedly fallen in love. I would point to the great consistency of the former sort of account amongst various instances thereof, likewise the latter, and finally to the clear correspondences and links between the two types of account.
Clearly—I would point out—there is something that sometimes happens to humans, in a certain class of situations, which alters their behavior in certain remarkably consistent ways, and the experience of which they report on (both at the time and after the fact) in remarkably consistent ways. That there is something there, is not disputable.
(This would suffice to establish the reality of the phenomenon, of course. However, I could also add some considerations from evolutionary psychology, pointing to reasons why “falling in love” should exist; and, likewise, findings from biology, neurology, etc., pointing to physiological underpinnings and manifestations of “falling in love”.)
To establish the value of falling in love would be another matter entirely.
My first observation would be to note that the point may well be moot. “If someday you fall in love,” I might say, “you won’t have any choice in the matter—it’ll simply happen to you, whether you think it’s got any value or not. Indeed, you may even find the experience unpleasant (although that is not the way I’d bet, given the experiences of most of humanity before you). But either way, your brain won’t ask for your opinion or permission—you’ll just be in love, and that’ll be that.”
You might ask, however, whether you ought to seek out situations which increase the likelihood that “falling in love” will happen to you. (After all, being in terror of your life is also an involuntary response, yet it makes sense to ask after its value, the better to decide whether or not to seek out tiger pits to climb into.)
Substantive answers to the question are likely to take two sorts of forms.
First, I could say that the experience of falling into (and being in) love is, itself, joyful, exciting, etc., in ways distinct from, and to a degree unmatched by, other human experiences. In other words, I could say that falling in love is worthwhile for its own sake. Of course, there isn’t any way I could convince you of that, but that’s not unusual; the same applies to the experience of eating ice cream, etc. This boils down to “try it; you’ll like it!”.
Second, I could say that falling in love is a necessary and inextricable part of being in love, which itself is intimately connected to having a deep and rewarding romantic/intimate relationship with another human being—in other words, that falling in love is just the first part of a process which (summed across all instances over a person’s lifetime) is likely to account for a significant chunk of the happiness, life satisfaction, joy, pleasure, etc., that one experiences in one’s lifetime.
The evidence for this, of course, is the experiences of other people. Much has been written on this, and I don’t know that I can add anything that isn’t trite… one thing I might say is that of those of my friends who’ve fallen in love, and entered into (and stayed in) long-term term relationships on the basis of that love, seem to be much happier than they were previously, and in particular, seem to make one another happy, in ways observable in ordinary, everyday interactions. (Of course one can have experiences and observe things that point to the opposite conclusion—but, again, this is too well-trod a topic to productively re-tread here.)
Now, all of that having been said, here’s a counter-question, before we get to kenshō: can you provide an analogous sort of answer, for “having a paranoid delusion” in place of “falling in love”? Does it exist? Does it have any value?
Now, all of that having been said, here’s a counter-question, before we get to kenshō: can you provide an analogous sort of answer, for “having a paranoid delusion” in place of “falling in love”? Does it exist? Does it have any value?
How on Earth is this relevant? I'm really not following you here. What do you hope to gain by having me try to grapple with this weird thing?
Maybe you're trying to… I don't know, get even with me for asking something you find absurd? Trying to defeat me in some kind of dual where you think I issued the first challenge? I really don't know.
In case it's not clear, the reason I asked you about the "falling in love" thing was to better understand what kind of thing "cake" as you mean it might even look like. It really does land as a type error to me. But if you could say "Oh, for falling in love, thus-and-such would be 'cake'", then I could go "Oh! Okay. Cool! So I think the analogy for kenshō might be XYZ. Does that work for you?" Then we could communicate.
The feel — and I could easily be misreading you here — but the feel I get from you here is like intellectual one-upmanship. Mental fencing.
If that's the case, please understand that I'm just not available for that. I will not engage with you at that level anymore.
If I've misread you, then please clarify what you're doing. I don't know how to orient to your tone here. If you meant it collaboratively, then please help me see how. I'd very much like to.
How on Earth is this relevant? I’m really not following you here. What do you hope to gain by having me try to grapple with this weird thing?
Well, if the relevance isn’t obvious, I think it will likely become obvious in the process of answering it. But, if not, I will certainly explain (what I see as) the relevance afterwards, in response to your answer.
In case it’s not clear, the reason I asked you about the “falling in love” thing was to better understand what kind of thing “cake” as you mean it might even look like.
Indeed, that is also why I asked my counter-question; both to explain, and to understand.
Maybe you’re trying to… I don’t know, get even with me for asking something you find absurd? Trying to defeat me in some kind of dual where you think I issued the first challenge? I really don’t know.
(ETA: I certainly don’t think that your question was absurd. If I did, I’d’ve said so, and not spent effort answering it!)
The feel — and I could easily be misreading you here — but the feel I get from you here is like intellectual one-upmanship. Mental fencing.
If that’s the case, please understand that I’m just not available for that. I will not engage with you at that level anymore.
I really don’t know what you mean by any of this (especially the “anymore” part, but really all of it).
If I’ve misread you, then please clarify what you’re doing. I don’t know how to orient to your tone here.
As I said—the point is to make clear what we’re talking about.
(I don’t think that it’s necessary to “orient to my tone”? In any case, generally speaking, if you assume that I mean just what I say, you won’t go far wrong.)
I really don’t know what you mean by any of this (especially the “anymore” part, but really all of it).
…
(I don’t think that it’s necessary to “orient to my tone”? In any case, generally speaking, if you assume that I mean just what I say, you won’t go far wrong.)
This is actually really clarifying. Thank you.
I now suspect there's a dimension of communication that's hyper-salient for me but invisible to you.
I won't try to convey that maybe invisible-to-you dimension here. I don't think that'd be helpful.
Instead I'll try to assume you have no idea what you're "saying" on that frequency. Basically that you probably don't mean things they way they implicitly land for me, and that you almost certainly don't consciously hold the tone I read in what you're saying.
That's as close as I can get to assuming that you "mean just what [you] say". Hopefully that'll smooth things out between us!
(ETA: I certainly don’t think that your question was absurd. If I did, I’d’ve said so, and not spent effort answering it!)
Okay, cool. Thanks for saying this!
> In case it’s not clear, the reason I asked you about the “falling in love” thing was to better understand what kind of thing “cake” as you mean it might even look like.
Indeed, that is also why I asked my counter-question; both to explain, and to understand.
I have to admit, I find this very confusing. I'm trying to understand what you mean by "cake". Maybe you were hoping to go "Here's 'cake' for falling in love. Now you try on this other topic, so I can thumbs-up or thumbs-down that you've understood what I mean by 'cake'." Is that it?
The thing is, I think I could provide a similar analysis, but I don't think it'd help me understand at all what you mean by "cake". That makes me pretty hesitant to spend the time and cognitive effort on producing that kind of matching analysis.
I now suspect there's a dimension of communication that's hyper-salient for me but invisible to you.
I won't try to convey that maybe invisible-to-you dimension here. I don't think that'd be helpful.
Instead I'll try to assume you have no idea what you're "saying" on that frequency. Basically that you probably don't mean things they way they implicitly land for me, and that you almost certainly don't consciously hold the tone I read in what you're saying.
That's as close as I can get to assuming that you "mean just what [you] say". Hopefully that'll smooth things out between us!
Okay, this is perhaps a complete side note, but this feels like a very precise pinpointing of what things like reduced affect and other of the most mysterious autistic communication difficulties can look like from the other (allistic, hyperfocused on emotional expressiveness, or otherwise very sensitive to affect) side.
From the perspective of folk with reduced affect, talking to people who rely strongly on affect, the experience strongly resembles that people they are interacting with are effectively listening to a random word generator rather than what they are saying. It is quite baffling and frustrating; especially since the explicit communication is often very carefully selected to communicate what they are trying to communicate.
So, basically, it's really good to recognize that this channel of communication can indeed hold random noise sometimes, and be aware of the extent to which you're focusing on it and the failure modes. (Presumably some of the times people have indeed corrected your perception of them.)
I don't think reduced affect necessarily corresponds (though might correlate) with an inability to discern things like emotional tone in other people, but it might be a bit trickier depending how much processing mirror neurons tend to handle. (I don't think anyone knows that, currently.)
I have to admit, I find this very confusing. I’m trying to understand what you mean by “cake”. Maybe you were hoping to go “Here’s ‘cake’ for falling in love. Now you try on this other topic, so I can thumbs-up or thumbs-down that you’ve understood what I mean by ‘cake’.” Is that it?
That would be a side benefit, certainly.
The thing is, I think I could provide a similar analysis, but I don’t think it’d help me understand at all what you mean by “cake”. That makes me pretty hesitant to spend the time and cognitive effort on producing that kind of matching analysis.
It need hardly be much effort. Not even as much as you’ve already spent on the last 2–3 comments in this thread, I’d say!
But if you could say "Oh, for falling in love, thus-and-such would be 'cake'"
The grandparent provides 700 words of this.
What do you hope to gain by having me try to grapple with this weird thing?
I read it as a warm-up question. Said provided what he considers a cake-like explanation of "falling in love." The obvious next step would be for him to ask you for an analogous explanation of kenshō. But if that were expected to go poorly, one might be tempted to try asking about something else, like "having a paranoid delusion", to exercise (or test) your ability to provide cake-like concreteness.
The three paragraphs following “Substantive answers to the question are likely to take two sorts of forms.”
Okay! Great, thank you.
This confirms I'm very thoroughly confused about what "cake" means to you here!
I thought you were looking for tangible proof of benefits, or something you could concretely try, or something like that. But now I know I have no idea what you're looking for!
I'll give examples to highlight my confusion. In your "cake" for falling in love, you say:
I could say that falling in love is worthwhile for its own sake. Of course, there isn’t any way I could convince you of that, but that’s not unusual; the same applies to the experience of eating ice cream, etc. This boils down to “try it; you’ll like it!”.
I seem to recall saying very similar things about kenshō. That there's something of deep importance, that this "insight" amounts to acknowledging it, that this is something you'd be super grateful for if it were to happen for you, and that there's not really much of a way for me to convince you of any of this. It's just a "Take a look and see for yourself" kind of thing.
That doesn't seem to have satisfied you. You still asked for "cake".
In particular, what you say here sounds to me like what I'd guess “assurances of having cake” would be.
In your second paragraph of "cake" you say:
falling in love is just the first part of a process which (summed across all instances over a person’s lifetime) is likely to account for a significant chunk of the happiness, life satisfaction, joy, pleasure, etc., that one experiences in one’s lifetime.
This lands for me as what I'd guess “allusions to kinds of cake” would be.
I could say something very similar about kenshō. I suspect I did in that monster thread five years ago. That if & when this flash of clarity comes online, there'll be a sense of something like "Oh holy fuck, I've been just living on autopilot! I haven't been alive here! I've been ignoring what actually goddamned matters just to tell myself some stories and live in fantasy! Whoa!!!" And it's very much just a beginning.
(There's a quote that goes something like "You have two lives. Your second life begins when you realize you have only one life." Kenshō is about beginning your second life.)
But again, this doesn't seem to have satisfied your need for "cake".
Your third paragraph includes:
those of my friends who’ve fallen in love, and entered into (and stayed in) long-term term relationships on the basis of that love, seem to be much happier than they were previously, and in particular, seem to make one another happy, in ways observable in ordinary, everyday interactions. (Of course one can have experiences and observe things that point to the opposite conclusion—but, again, this is too well-trod a topic to productively re-tread here.)
So, on this regarding kenshō I've been maybe too vague. Very attuned to the ability of folk to point at evidence of the opposite conclusion.
But if I can make a similar caveat as you've made here, I think I can point pretty clearly at this.
The people I know who are on the other side of this are alive. Engaged. Interesting. They're themselves much more deeply. More interested in really playing the game of life. Less willing to tolerate bullshit, especially in their for-fun social interactions.
They also almost all have war stories involving the collapse after awakening. A lot of lies people live can't work once they admit to themselves that they're lies. And it's hard not to admit stuff like that in the midst of an explosion of Light, at least in my experience. Breakups, financial collapse, and physical illness are not uncommon. It's usually temporary and most of them say that they totally wouldn't have it any other way — at least once they're through the other side. Some do get stuck. And there's a potential survivorship bias here in my account.
It'd be weird to call that a benefit. It's more like an attribute I notice over and over again. I totally had that. Arguably I still do: it feels like a deep existential allergy to all lies and bullshit turned on deep in my core, and now I'm on a lifelong journey for total and ever-perfecting alignment with… well, truth.
But as things go in terms of "What can we see this actually doing in the world?", here are a few attributes. The Dark Night stuff can be awful to go through, but it's like the vomiting part of food poisoning. It's not like the point of that is to be enjoyable, but you still want to have done it.
Now, if my saying all that still doesn't count for you as "cake"… then I have no idea how to proceed. You're going to have to define what you're looking for differently if you want me to have any chance of answering you on this point.
Well, let’s recap a bit. You wrote:
If you were trying to convey this whole “falling in love” thing to me, while I’m suspicious about whether it exists or that if it exists that it has any value, and I were pressing you for “cake” about “falling in love”, what would you offer?
This is a reasonable enough question, as I said, but it does bear noting that it’s not like I’ve actually written any posts about how great “falling in love” is and how people should try doing it, etc. (I’m not even sure I would actually advocate for falling in love, if you asked me whether I think that you should try to do it, and were skeptical about it!)
Now, you asked what I’d offer if I were trying to convey “falling in love” and were asked for “cake” (i.e., answers to “what’s it good for?”), and I answered “here’s what I’d offer”. That response wasn’t very substantive! To abuse the metaphor somewhat, if we imagine our metaphorical cake as, say, tiramisu, my response would be, perhaps, a single ladyfinger dipped in coffee liqueur, with a dollop of whipped cream on it—not really a whole cake, with all the ingredients in place, fully assembled and finished with all the details, but more like a proof-of-concept, establishing that the basic idea works and is essentially sensible. (You could also call it a sketch rather than a finished portrait, or use any number of similar metaphors.)
That having been said, let’s move to the non-metaphorical object level:
I thought you were looking for tangible proof of benefits, or something you could concretely try, or something like that. But now I know I have no idea what you’re looking for!
Tangible proof of benefits is good, but concrete description of benefits is the thing that’s got to come first. Otherwise, what’s being proven?
As far as “something to concretely try”, please note that this is basically of no value unless either (a) I can have some reasonable expectations for what sort of thing I’ll get if I try it, or (b) trying is costless or close to it. Otherwise, it’s little more than a bluff.
I could say that falling in love is worthwhile for its own sake. Of course, there isn’t any way I could convince you of that, but that’s not unusual; the same applies to the experience of eating ice cream, etc. This boils down to “try it; you’ll like it!”.
I seem to recall saying very similar things about kenshō. That there’s something of deep importance, that this “insight” amounts to acknowledging it, that this is something you’d be super grateful for if it were to happen for you, and that there’s not really much of a way for me to convince you of any of this. It’s just a “Take a look and see for yourself” kind of thing.
That doesn’t seem to have satisfied you. You still asked for “cake”.
Well, indeed. As I said, there isn’t any way I could convince you that love is worthwhile for its own sake if you didn’t already believe it. Nor should you be convinced! You would be quite right to disbelieve me! (Especially because if you didn’t believe that there’s any value to falling in love, that would be evidence that you are the the sort of person for whom there isn’t any value to falling in love.)
There’s two things to note here.
One is that you’ve certainly claimed all sorts of benefits to kenshō, so “worthwhile for its own sake” isn’t all that’s being claimed; if it were, we’d have had (and would be having) a very different conversation.
The other is that the ice cream example is illustrative. Trying ice cream is something you can do for trivial amounts of money and effort, and which has basically no downsides (assuming you’ve checked the ice cream’s ingredients for known allergens you’re sensitive to, etc.). Notice that most people (quite reasonably) still don’t try everything that they could try with comparably little effort/cost expenditure, simply because there’s so many such things! But if a thing in this category is brought to your attention, by someone whose word you have at least some minimal reason not to distrust—well, why not, right? It’s costs so little.
It hardly needs pointing out that this is absolutely not the case for falling in love… and still less so for kenshō.
falling in love is just the first part of a process which (summed across all instances over a person’s lifetime) is likely to account for a significant chunk of the happiness, life satisfaction, joy, pleasure, etc., that one experiences in one’s lifetime.
This lands for me as what I’d guess “allusions to kinds of cake” would be.
Just so! Which is why my answer was a sketch of the sorts of things I would say if I were trying to convince you of the value of “falling in love”. If I were trying to construct the finished portrait, so to speak, I’d have to go into considerably more detail, offer actual examples and specific accounts, etc., etc.
And that’s the problem with the rest of what you write here: it’s “allusions to…”, but where are the specifics?
Here’s another approach: on the old Kensho post, our discussion broke off with a couple of my comments having been left unanswered. One very relevant comment was this one, where I ask for specifics about a number of vague things you wrote in a previous comment. Perhaps you could give those specifics now?
Why don't you care for the "looking" framing anymore?
Also, do you have an example of: "getting better at attending to what matters to you, instead of attending to what you think matters to you, when there's a difference".
Would a central example be someone focusing on trying to become popular, not realising that it is only instrumental towards feeling positive about themselves?
do you have an example of: "getting better at attending to what matters to you, instead of attending to what you think matters to you, when there's a difference".
Would a central example be someone focusing on trying to become popular, not realising that it is only instrumental towards feeling positive about themselves?
I wouldn't say that's a central example. But I think it's a good one. Simple, clear, easy to access.
A much bigger and more central example to me is death. Most people don't seem to have clearly seen their own mortality. They're thinking about their own deaths like a Cartesian agent (i.e., not embedded — like the Alexei robot in this post). They know death happens to everyone eventually, and that it'll happen to "them", but it's almost like they're talking about a video game character from the outside. Like "Yep, at some point my little Mario figure there will stop when we turn off the gaming console."
Something really big shifts when folk get a terminal diagnosis. It's not just "Oh, I thought I had years, but now I have months/weeks/etc." There's a grappling-with that's hard to convey to non-terminal folk. "The end" becomes subjectively real. Like "Oh shit, I'm going to experience this from behind my eyes, inside this skin." It can feel very lonely and alienating, both because of the existential horror of the situation, but also because it becomes super clear how others are running strange scripts that just don't make sense.
I include most immortalists and transhumanists in this. I'm not talking just about "Are you taking practical actions to deal with your mortality?" I mean that there's a deep existential thing to grapple with. Even true immortals would have to orient to it: If you never ever die, then that means you're facing eternity. That's like something from behind your eyes.
(This is something even the literalist Christian mythos doesn't properly address AFAICT: Great, you go to Heaven. Then what? Do you feel time passing? What's that like? Goes on literally forever? Or does it… fade away at some point? Or do you leave time when you enter Heaven — in which case what's the subjective experience of that? This isn't just abstract philosophy. It's a damn meaningful question!)
It turns out that most things don't matter in the face of this. The whole thing with deathbed regrets amounts to "I'm sorry I didn't see this sooner and take it seriously while I still had time."
But it's not something you're likely to really get just by listening to elders and taking their advice. "Work less, connect with family more." That's good, and you'll be grateful for that in the end! But you won't understand why until you've really truly seen your own death clearly.
To the degree you do, it just won't be tempting anymore to work instead of connect (for instance).
But that's a really huge example. Maybe one of the biggest. Maybe maybe the biggest.
I do think the "Oh, I don't actually care about being popular" thing you suggest is pretty good. It has the right quality of "Now that I've seen it, I can't un-see it, and I've been seriously wasting my time on this."
I just wanted to flesh this out a bit, to hint at the scope.
Hopefully that made some sense.
Why don't you care for the "looking" framing anymore?
I'm honestly not sure. Something sits wrong about it. Some initial stabs at the intuition are:
- It feels too binary. Like you're either Looking or you're not. That doesn't seem right.
- I think it might be several skills/senses folded into one. Or maybe something like a sense generator. (Lots of folk have trouble with conscious interoception, and that can be changed over time via a move I might have called "Looking" before.)
- In practice, the relevant thing to develop seems to be something like capacity to receive the information rather than forcing oneself to seek it. I think intentionally seeing something beyond the usual ontology ("Looking") emerges naturally from being both willing and capable, once you know what direction to point your attention in.
But the overall feel is something like… the whole thing makes it sound pompous. Like "Oho! Here's this special magical skill!"
I mean, it is. It's quite potent. I'll sometimes mythically refer to this as "Mage Sight", or break it down into subtypes of Sight (like Spirit Sight or Prime Sight).
But I think something screwy happens when there's a social tone along the lines of "I see something you don't! 'Cause I'm bothering to look at all!"
Even when there's some truth to that, it's unkind. It doesn't honor the reason why the person hasn't pointed their attention there yet.
Also, in many cases I can just say "You want to see? Okay. Look here, then here, then here." And something clicks for my listener. Sometimes.
So, no reason to make the act of looking there something special. It's just perception.
Also, in many cases I can just say "You want to see? Okay. Look here, then here, then here." And something clicks for my listener.
I assume the "here"s are placeholders for some specific things you would be saying to a particular listener. Can you give some examples of things that those places have held?
Can you give some examples of things that those places have held?
Sure. I'll give just one for now. These take a while to name in writing, at least they way they occur to me.
Here's a recurring one: I'll be talking with someone in a coaching session, and I'll pick up on how they're "adding extra".
This is something that's easy to point out in live conversation or over video but I find tricky in writing. It's a tone thing. If I look at the cup next to me and note "This is a cup", I'm simply noting. There's nothing extra. But I can add extra with an emotional tone of "I keep this cup next to me to hydrate myself, because I take care of myself, which is something GOOD I do." I think this is easy to hear with some practice even if the words are identical ("This is a cup").
Usually when someone is adding extra, they have some unrecognized pain. Most of the time this pain roots down in a universal thing — something I've come to call "the pain of duality". It's a basic split from something core. It's roughly the same in everyone best as I can tell, but each person sort of holds and experiences it in their own way.
So when I see someone adding extra, and they've asked for my guidance, I'll sometimes guide them to awareness of this core pain:
- "Here it seems to me that you're saying XYZ [like "I need to finish my thesis"], but you're also adding something extra. It occurs to me as a tone of ABC [usually like there's something wrong with them, or that their value is based on something external, or that something is existentially wrong]. There's nothing wrong with that. I just imagine it's uncomfortable. Do you see what I'm pointing at?"
- We do some calibration, and I adjust based on their feedback (like if I was missing them in some key way). Then if I still see this core pain in them and they agree I'm seeing them clearly, on to the next step:
- "Okay. Now, for me, in this spot I tend to feel PQR [something like "afraid something will go wrong if I don't take care of this task"]. But that's how the energy feels when it hits my thinking. Underneath that is something wordless. More like a creeping feeling, like reality itself is unsafe or unreliable."
- The point here is to give an example of what it means to feel the energy behind something in consciousness. If my first example doesn't click for them, I'll give a few others.
Usually they either notice the core pain or adjust my perception of them. More often the former. It tends to result in a direct kind of seeing, the same way you can directly "see" the feeling of your tongue in your mouth: it's somehow more unmediated than thoughts about the thing are.
When I wrote Kenshō I might have called this "Looking at your soul pain". It's about seeing more directly instead of just thinking about mental models of the thing.
I just think that reifying Looking does something odd to this process. It's just noticing what you experience when you look where someone is pointing. Even if that someone is yourself.
Hopefully that's somewhat clear. With more time & effort I might have come up with a simpler example. ("Sorry this letter is so long, I didn't have time to write a shorter one.")
Thank you. This is cake! And not only cake, but a cake I have tasted before, although perhaps made to a variant of the recipe. I'm familiar with people adding on their own "stuff" to the things that happen. Personally, I would put less emphasis on their emotional reaction as the thing of importance than whatever they are believing in that moment, that the emotion is a reaction to. If the reaction is dysfunctional, unearthing the beliefs and tracing them to their origins in past events can be helpful in dissolving it.
I generally don't and wouldn't expect people to increase understanding or epistemology based on meditation. I would expect productivity gains in some cases, depending on how they do it, and happiness gains in many cases.
More speculatively, I think the risk of degrading your epistemology is probably low if you go in with a sufficiently skeptical mindset, which you seem to have.
As a Christian, I'm not surprised you notice such a phenomenon. Meditation opens you up spiritually to external influence. Not just epistemically, but ontologically. Meditation gives external things influence over yourself to the framework level. This is why Christians meditate with the most powerful spirit (Holy Spirit) so that we don't run into issues such as incorrect "programming" or "misalignment" or "over-fitting". The complete form of meditation is commonly called prayer to differentiate it from incomplete forms of meditation.
In my experience with meditation literature, and with the sort of tech bro who takes acid, becomes enlightened in some way and becomes a metta bro, the "ontologies" or explanatory frameworks that they use are most always very bad and deeply incorrect and in some cases can lead people to very dark places for a long time. However they only provide interpretation and articulation of insight, or practice instructions for finding the insight in the first place, and are not the same as that insight. I do think the various insights are still valuable, and that they are necessary (but clearly quite insufficient) for doing good philosophy (eg. finding better explanatory frameworks, among many other things).
I would say that the main benefits of enlightenment are that I have cleared out a bunch of inefficient wasted motion in my mind, a sort of mental sludge, and I have direct access to a bunch of tools for more directly working with my emotions, motivations, beliefs and thoughts and as a consequence I am a lot more sane than I was six years ago when all that was fresh. Sane in the sense of being more emotionally regulated, and a more moral person, which is more important to me, but I would also claim in the classic LW sense of having less wrong beliefs as well I guess
I also now have a philosophy that is actually practically useful in my day to day life, doesn't make me miserable, and doesn't function as a distraction, and meditation was necessary for this. However it is somewhere so alien to LW that you'd probably lump me in with the very bad and deeply incorrect philosophies. However, I think it was because I had already moved somewhere else from LW philosophy before getting deeper into meditation that I ended up with a partial philosophy which is roughly correct on the parts that it covers. I think the scientismic "ontologies" in which LW's is included directly lead to bad places in their interaction with meditation.
So idk. I think it's good, I don't know how to direct you towards the path that I went down unless you want to engage in tens of thousands of words between us, or in a similar amount of words in works of philosophy. I find the literature unsatisfying and I'm bad at writing.
There are risks, and it's also necessary if you want to go down certain paths seeking knowledge. ymmv
I get the impression from talking to people who work professionally on reducing ai x-risk (largely taken, would include doing ops for FLI) that less than half of them could give an accurate and logically sound model of where ai x-risk comes from and why it is good that they do the work they do (bonus points if they can compare it to other work they might be doing instead). I'm generally a bit disappointed by this because to me it doesn't seem that hard to get everyone who's a professional knowledgeable on the basics, and it seems worthwhile as more people could be autonomous in assessing strategic decisions and making sure implementation of plans serves the right aims.
An example of why having a model of x-risk and directions for reducing it matters : Imagine you're an ops person with money to organize a half-day conference. What do?
What city to choose, which participants to invite?
If you have no model of anything, I guess you can copycat - do a small scale EAG-like event for AI Safety people, or do a scientific conference like thing. That's okay, probably not a total waste of money (except to the extent you don't know ai safety theory, it's possible to do counter-productive actions like giving a platform to the wrong people).
Imagine you have the following simple model :
- AI is being improved by people
- If deployed with the current alignment tech and governance level, powerful autonomous AGI could have convergent instrumental goals that lead to takeover (sharp left turn models) or large scale deployment of AGI in society can eek out human influence (Out with a Whimper or Multipolar Failure)
- Thus we should do actions that get us the best alignment tech and governance level by the time anyone is able to train and deploy dangerous IA systems.
- This means we could do actions that
- - Slow down dangerous AI development
- - Accelerate alignment tech development
- - Accelerate getting a good level of governance
Each of these need sub-models (all interlinked), let's detail a simple one for governance
Things that accelerate getting a good level of governance
- Better knowledge and models of ai x-risk (scenario+risk planning), demonstrations of precursors (model organisms of misalignment, ..), increasing scientific consensus about the paths to avoid
- Spreading the above knowledge to the general public (to have politicians be supported in economy-limiting policy)
- Spreading the above knowledge to policy-makers
- Having technically competent people write increasingly better policy proposals (probably stuff leading to international regulation of dangerous AI)
Now that you have these, it's much easier to find a few key metrics to optimize for your ops event. Doing it really well might include you talking to other event organizers to know what has and hasn't been done, what works, what's been neglected etc, but even without all that you can decide to act on:
Improvement in attendees' knowledge and models of AI x-risk
Pre-event and post-event surveys or informal conversations to gauge attendees' understanding of AI risk scenarios and mitigation strategies.
Structure the event with interactive sessions explicitly designed to clarify AI x-risk models, scenario planning, and concrete policy implications.
Potential reach for disseminating accurate AI safety knowledge
Choose guests who have strong networks or influence with policy-makers, academics, or public media.
Select a location close to influential governmental bodies or major media centers (e.g., Washington D.C., Brussels, or Geneva).
(I realize now that I wrote a full example that this might have been a mini post to serve as short reference, a wireframe for thinking through this, please reply if you think a cleaned up version of this would be good)
To get back to my original point, it's currently my impression that many who work in AI x-risk reduction, for example in ops, could not produce a "good" version of this above draft even with 2 hours of their time because of lacking background knowledge. I hope the above example sufficiently illustrates that they could be doing better work if they did.
There's tacit knowledge in bay rationalist conversation norms that I'm discovering and thinking about, here's an observation and related thought. (I put the example later after the generalisation because that's my preferred style, feel free to read the other way).
Willingness to argue righteously and hash out things to the end, repeated over many conversations, makes it more salient when you're going for a dead end argument. This salience can inspire you to do argue more concisely and to the point over time.
Going to the end of things generates ground data on which to update your models of arguing and conversation paths, instead of leaving things unanswered.
So, though it's skilful to know when not to "waste" time on details and unimportant disagreements, the norm of "frequently enough going through til everyone agrees on things" seems profoundly virtuous.
Short example from today, I say "good morning". They point out it's not morning (it's 12:02). I comment about how 2 minutes is not that much. They argue that 2 minutes is definitely more than zero and that's the important cut-off.
I realize that "2 minutes is not that much" was not my true rebuttal, that this next token my brain generated was mostly defensive reasoning rather than curious exploration of why they disagreed with my statement. Next time I could instead note they're using "morning" to have a different definition/central cluster than I, appreciate that they pointed this out, and decide if I want to explore this discrepancy or not.
Many things don't make sense if you're just doing them for local effect, but do when you consider long term gains. (something something naive consequentialism vs virtue ethics flavored stuff)
Self modeling is a really important skill, and you can measure how good you are at it by writing predictions about yourself. (Modelling A notably important one for people who have difficulty with motivation is predicting your own motivation - will you be motivated to do X in situation Y?
If you can answer that one generally, you can plan to actually anything you could theoretically do, using the following algorithm : from current situation A, to achieve wanted outcome Z, find a predecessor situation Y from which you'll be motivated to get to Z (eg. have written 3 paragraphs of 4 of an essay), and a predecessor situational X from which you'll get to Y, iterate til you get to A (or forward chain, from A to Z). Check that indeed you'll be motivated each step of the way.
How can the above plan fail? Either you were mistaken about yourself, or about the world. Figure out which and iterate.
Some people have short ai timelines based inner models that don't communicate well. They might say "I think if company X trains according to new technique Y it should scale well and lead to AGI, and I expect them to use technique Y in the next few years", and the reasons for why they think technique Y should work are some kind of deep understanding built from years of reading ml papers, that's not particularly easy to transmit or debate.
In those cases, I want to avoid going into details and arguing directly, but would suggest that they use their deep knowledge of ML to predict existing recent results before looking at them. This would be easy to cheat, so I mostly suggest this for people to check themselves, or check people you trust to be honorable. Concretely, it'd be nice if when some new ml paper with a new technique comes out, someone compilés a list of questions answered by that paper (eg is technique A better than technique B for a particular result) and posts it to LW so people can track how well they understand ML, and thus (to some extent) short timelines.
For example a recent paper examinés how data affects performance on a bunch of benchmarks, and notably tested training either on an duplicated dataset (a bunch of common crawls), or deduplixated (the same except remove same documents that were shared between crawls). Do you expect deduplication in this case raises or lowers performance on benchmarks? If we could have similar questions when new results come out it's be nice.
Curated and popular this week