Lukas_Gloor

Wiki Contributions

Comments

My comment above was mostly coming from a feeling of being upset, so I'm writing a second comment here to excavate why I feel strongly about this (and decide whether I stand by it on reflection).

I think the reason I care about this is because I'm concerned that we're losing the ability to distinguish people who are worth learning from ("genuine experts") from people who have a platform + an overconfident personality. With this concern in mind, I don't want to let it slide that someone can lower the standards of discourse to an arbitrary degree without suffering a loss of their reputation. (I would say the same thing about some AI safety advocates.) Of course, I agree it reflects badly on AI safety advocates if they're needlessly making it harder for critics to keep an open mind. Stop doing that. At the same time, it also reflects badly on Meta and the way the media operates ("who qualifies as an expert?") that the chief AI scientist at the company and someone who gets interviewed a lot has some of the worst takes on the topic I've ever seen. That's scary all by itself, regardless of how we got here.

I think you're giving LeCun way too much credit if you're saying his arguments are so bad now because other people around him were hostile and engaged in a low-quality way. Maybe those things were true, but that doesn't excuse stuff like repeating bad arguments after they've been pointed out or confidently proclaiming that we have nothing to worry about based on arguments that obviously don't hold up. 

There's a difference between "having a desire for limerence" and "being the person capable of developing limerence." Some people may not have a desire for it, but they get limerent pretty quickly with the right triggers. (Some people may even hate the fact that their brain does this because it keeps getting them into bad situations, but they keep developing limerence and are a slave to it.)

This led me to research it, and it seems like limerence is a highly culture specific, and is likely more a cultural meme than an emotion inherent to human brains.

It's a distinct emotional state comparable to being on a powerful drug. So, it can't just be a cultural meme. Of course, it could be that the frequency with which the emotional state is elicited is culture-dependent. (Just like some culture have a higher/lower prevalence of depression.)What's also culture-dependent is whether you romanticize limerence or whether you look at it as something dysfunctional. As you mention, some people seem to think good romance requires limerence. I think that's irrational (unless you care more about the "hedonics" of being in love than finding someone actually compatible). 

I agree that there's a connection from limerence to drama – though this is for indirect correlational reasons rather than limerence being defined through drama.

If I were to guess, I would say limerence is a side effect of the emotional and sexual frustration of the young and inexperienced humans who dabble in their first relationships,

I suspect the same thing, I think it might have to do with unmet needs and the fantasy of fulfilling them all through this one ideal person you met (who you don't really know yet, but you're projecting onto them everything that can fix your loneliness/pain). (It can be completely non-sexual). What I don't understand is how you go from the description "side effect from frustration" to "it's a cultural meme." Depression is an emotional state that we could describe as being a side effect of unmet needs as well, but this doesn't make it a cultural meme. 

I think there also might be a lot genetic variation to people's propensity to develop limerence?

This could be true as a reason why some people de-prioritize s-risks, but I don't think it's a correct statement.  See the section "s-risk reduction is separate from alignment work" here.  

Related to the "personal fit" explanation: I'd argue that the skills required to best reduce s-risks have much overlap with the skills to make alignment progress (see here).  

At least, I think this goes for directly AI-related s-risks, which I consider most concerning, but I put significantly lower probabilities on them than you do.

For s-risks conditioned on humans staying in control over the future, we maybe wouldn't gain much from explicitly modelling AI takeoff and engaging in all the typical longtermist thought. Therefore, some things that reduce future disvalue don't have to look like longtermism? For instance, common sense ways to improve society's rationality, coordination abilities, and values. (Maybe there's a bit of leverage to gain from thinking explicitly about how AI will change things.) The main drawback to those types of interventions is (1) disvalue at stake might be smaller than the disvalue for directly AI-related s-risks conditional on the scenarios playing out, and (2) it only matters how society thinks and what we value if humans actually stay in control over the future, which arguably seems pretty unlikely.

Otherwise you have a constant large compute overhang.


I think we should strongly consider finding a way of dealing with that rather than only looking at solutions that produce no overhang. For all we know, total compute required for TAI (especially factoring in future algorithmic progress) isn't far away from where we are now. Dealing with the problem of preventing defectors from exploiting a compute overhang seems potentially easier than solving alignment on a very short timescale. 

Said's way of asking questions, and the uncharitable assumptions he sometimes makes, is one of the most off-putting things I associate with LW. I don't find it okay myself, but it seems like the sort of thing that's hard to pin down with legible rules. Like, if he were to ask me "what is it that you don't like, exactly" – I feel like it's hard to pin down.

Edit: So, on the topic of moderation policy, seems like the option that individual users can ban specific other users if they have trouble dealing with their style or just if conflicts happen, that seems like a good solution to me. And I don't think it should reflect poorly on the banner (unless they ban an extraordinary number of other users). 

I like the reasoning behind this post, but I'm not sure I buy the conclusion. Here's an attempt at excavating why not:

If I may try to paraphrase, I'd say your argument has two parts:

(1) Humans had a "sharp left turn" not because of some underlying jump in brain capabilities, but because of shifting from one way of gaining capabilities to another (from solo learning to culture).

(2) Contemporary AI training is more analogous to "already having culture," so we shouldn't expect that things will accelerate in ways ML researchers don't already anticipate based on trend extrapolations.

Accordingly, we shouldn't expect AIs to get a sharp left turn.

I think I buy (1) but I'm not sure about (2). 

Here's an attempt at arguing that AI training will still get a "boost from culture." If I'm right, it could even be the case that their "boost from culture" will be larger than it was for early humans because we now have a massive culture overhang.

Or maybe "culture" isn't the right thing exactly, and the better phrase is something like "generality-and-stacking-insights-on-top-of-each-other threshold from deep causal understanding." If we look at human history, it's not just the start of cultural evolution that stands out – it's also the scientific revolution! (A lot of cultural evolution worked despite individual humans not understanding why they do the things that they do [Henrich's "The Secret of our Success] – by contrast, science is different and requires at least some scientists to understand deeply what they're doing.)

My intuition is that there's an "intelligence" threshold past which all the information on the internet suddenly becomes a lot more useful.  When Nate/MIRI speak of a "sharp left turn," my guess is that they mean some understanding-driven thing. (And it has less to do with humans following unnecessarily convoluted rules about food preparation that they don't even understand the purpose of, but following the rules somehow prevents them from poisoning themselves.) It's not "culture" per se, but we needed culture to get there (and maybe it matters "what kind of culture" – e.g., education with scientific mindware).

Elsewhere, I expressed it as follows (quoting now from text I wrote elsewhere):

I suspect that there’s a phase transition that happens when agents get sufficiently good at what Daniel Kokotajlo and Ramana Kumar call “P₂B” (a recursive acronym for “Plan to P₂B Better”). When it comes to “intelligence,” it seems to me that we can distinguish between “learning potential” and “trained/crystallized intelligence” (or “competence”). Children who grow up in an enculturated/learning-friendly setting (as opposed to, e.g., feral children or Helen Keller before she met her teacher) reach a threshold where their understanding of the world and their thoughts becomes sufficiently deep to kickstart a feedback loop. Instead of aimlessly absorbing what’s around them, they prioritize learning the skills and habits of thinking that seem beneficial according to their goals. In this process, slight differences in “learning potential” can significantly affect where a person ends up in their intellectual prime. So, “learning potential” may be gradual, but above a specific threshold (humans above, chimpanzees below), there’s a discontinuity in how it translates to “trained/crystallized intelligence” after a lifetime of (self-)directed learning. Moreover, it seems that we can tell that the slope of the graph (y-axis: “trained/crystallized intelligence;” x-axis: “learning potential”) around the human range is steep.

To quote something I’ve written previously:

“If the child in the chair next to me in fifth grade was slightly more intellectually curious, somewhat more productive, and marginally better dispositioned to adopt a truth-seeking approach and self-image than I am, this could initially mean they score 100%, and I score 95% on fifth-grade tests – no big difference. But as time goes on, their productivity gets them to read more books, their intellectual curiosity and good judgment get them to read more unusually useful books, and their cleverness gets them to integrate all this knowledge in better and increasingly more creative ways. [...] By the time we graduate university, my intellectual skills are mostly useless, while they have technical expertise in several topics, can match or even exceed my thinking even on areas I specialized in, and get hired by some leading AI company.

[...]

If my 12-year-old self had been brain-uploaded to a suitable virtual reality, made copies of, and given the task of devouring the entire internet in 1,000 years of subjective time (with no aging) to acquire enough knowledge and skill to produce novel and for-the-world useful intellectual contributions, the result probably wouldn’t be much of a success. If we imagined the same with my 19-year-old self, there’s a high chance the result wouldn’t be useful either – but also some chance it would be extremely useful. [...]  I think it’s at least plausible that there’s a jump once the copies reach a level of intellectual maturity to make plans which are flexible enough [...] and divide labor sensibly [...].”

In other words, I suspect there’s a discontinuity at the point where the P₂B feedback loop hits its critical threshold.

So, my intuition here is that we'll see phase change once AIs reach the kind of deeper understanding of things that allows them to form better learning strategies. That phase transition will be similar in kind to going from no culture to culture, but it's more "AIs suddenly grokking rationality/science to a sufficient-enough degree that they can stack insights with enough reliability to avoid deteriorating results." (Once they grok it, the update permeates to everything they've read – since they read large parts of the internet, the result will be massive.)

I'm not sure what all this implies about values generalizing to new contexts / matters of alignment difficulty. You seem open to the idea of fast takeoff through AIs improving training data, which seems related to my notion of "AIs get smart enough to notice on their own what type of internet-text training data is highest quality vs what's dumb or subtly off." So, maybe we don't disagree much and your objection to the "sharp left turn" concept has to do with the connotations it has for alignment difficulties.

"Effective compute" is the combination of hardware growth and algorithmic progress? If those are multiplicative rather than additive, slowing one of the factors may only accomplish little on its own, but maybe it could pave the way for more significant changes when you slow both at the same time? 

Unfortunately, it seems hard to significantly slow algorithmic progress. I can think of changes to publishing behaviors (and improving security) and pausing research on scary models (for instance via safety evals). Maybe things like handicapping talent pools via changes to immigration policy, or encouraging capability researchers to do other work. But that's about it. 

Still, combining different measures could be promising if the effects are multiplicative rather than additive. 

Edit: Ah, but I guess your point is that even a 100% tax on compute wouldn't really change the slope of the compute growth curve – it would only move the curve rightward and delay a little. So we don't get a multiplicative effect, unfortunately. We'd need to find an intervention that changes the steepness of the curve.   

At first maybe you try to argue with them about it. But over time, a) you find yourself not bothering to argue with them

>Whose fault is that, exactly…?

b) even when you do argue with them, they’re the ones choosing the terms of the argument.

>Ditto.

If they think X is important, you find yourself focused on argue whether-or-not X is true, and ignoring all the different Ys and Zs that maybe you should have been thinking about.

>Ditto.

---

I agree that nothing about the examples you quote is unacceptably bad – all these things are "socially permissible." 

At the same time, your "Whose fault is that, exactly...?" makes it seem like there's nothing the guru in question could be doing differently. That's false.

Sure, some people are okay with seeing all social interactions as something where everyone is in it for themselves. However, in close(r) relationship contexts (e.g. friendships, romantic relationships, probably also spiritual mentoring from a guru?), many operate on the assumption that people care about each other and want to preserve each other's agency and help each other flourish. In that context, it's perfectly okay to have an expectation that others will (1) help me notice and speak up if something doesn't quite feel right to me (as opposed to keeping quiet) and (2) help me arrive at informed/balanced views after carefully considering alternatives, as opposed to only presenting me their terms of the argument.

If the guru never says "I care about you as a person," he's fine to operate as he does. But once he starts to reassure his followers that he always has their best interest in mind – that's when he crosses the line into immoral, exploitative behavior. 

You can't have it both ways. If your answer to people getting hurt is always "well, whose fault was that?" 

Then don't ever fucking reassure them that you care about them!

In reality, I'm pretty sure "gurus" almost always go to great lengths convincing their followers that they care more about them than almost anyone else. That's where things become indefensible.

Load More