All of Vaniver's Comments + Replies

Yudkowsky and Christiano discuss "Takeoff Speeds"

your point is simply that it's hard to predict when that will happen when you just look at the Penn Treebank trend.

This is a big part of my point; a smaller elaboration is that it can be easy to trick yourself into thinking that, because you understand what will happen with PTB, you'll understand what will happen with economics/security/etc., when in fact you don't have much understanding of the connection between those, and there might be significant discontinuities. [To be clear, I don't have much understanding of this either; I wish I did!]

For example, ... (read more)

Vaniver's Shortform

My boyfriend is getting into Magic, and so I've been playing more of it recently. (I've played off and on for years; my first set was Tempest, which came out in October 1997). The two most recent sets have been set in Innistrad, their Gothic Horror setting.

One of the things I like about Innistrad is that I think it has the correct red/black archetype: Vampires. That is, red/black is about independence and endorsed hedonism; vampires who take what they want from those weaker than them, and then use the strength gained from that taking to maintain their stat... (read more)

Yudkowsky and Christiano discuss "Takeoff Speeds"
Vaniver3d49Ω20

it seems like extrapolating from the past still gives you a lot better of a model than most available alternatives.

My impression is that some people are impressed by GPT-3's capabilities, whereas your response is "ok, but it's part of the straight-line trend on Penn Treebank; maybe it's a little ahead of schedule, but nothing to write home about." But clearly you and they are focused on different metrics! 

That is, suppose it's the case that GPT-3 is the first successfully commercialized language model. (I think in order to make this literally true you... (read more)

7Matthew Barnett3dI think it's the nature of every product that comes on the market that it will experience a discontinuity from having zero revenue to having some revenue at some point. It's an interesting question of when that will happen, and maybe your point is simply that it's hard to predict when that will happen when you just look at the Penn Treebank trend. However, I suspect that the revenue curve will look pretty continuous, now that it's gone from zero to one. Do you disagree? In a world with continuous, gradual progress across a ton of metrics, you're going to get discontinuities from zero to one. I don't think anyone from the Paul camp disagrees with that (in fact, Katja Grace talked about this [https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/#Starting_high] in her article). From the continuous takeoff perspective, these discontinuities don't seem very relevant unless going from zero to one is very important in a qualitative sense. But I would contend that going from "no revenue" to "some revenue" is not actually that meaningful in the sense of distinguishing AI from the large class of other economic products that have gradual development curves.
[linkpost] Why Going to the Doctor Sucks (WaitButWhy)

Do you have any thoughts on what makes someone a spike?

I mean, the OP has two examples of the target customer: someone with a serious autoimmune disorder, and someone who got breast cancer while young. 

My interest in this sort of thing stems from having low energy compared to people around me, and wouldn't be surprised if there's a medical treatment available that somehow increases my productive hours by 20-100%. Compare to my boyfriend, who already works >60hr weeks, where I would be astounded if a similar intervention existed for him.

2% productiv

... (read more)
[Book Review] "Sorceror's Apprentice" by Tahir Shah

You have to do it quickly, your hand has to be perfectly dry and you have to not splash yourself.

Also, if this is anything like other tricks like this, you probably don't want to have any hairs on your hand.

[linkpost] Why Going to the Doctor Sucks (WaitButWhy)

I think if you're earning 100k/yr, it's pretty likely that improvements to your health can make you more than 2% more productive, and thus it's worth it. Unfortunately I think this is pretty spiky, where some people will get >10% improvements and many people will get 0% improvements, and it may be possible to tell ahead of time which group you'll be in. [Like, the relevant factor shouldn't be "I'm young" but something more like "I love the experience of being in my body."]

1Josh Jacobson5dI’m quite skeptical that improvements will be realized by this methodology. Not clear that there are health improvement gains in expectation.
2adamzerner5dThe spiky part makes sense. I'm the type of person who is very accepting of expected value based reasoning, so I'm not turned off by it. Although it does beg the question of whether I am the type of person who is a spike or not. I don't feel like I am, but maybe there's something I'm overlooking. Do you have any thoughts on what makes someone a spike? 2% productive seems possible, but I don't really have a good sense of what that number is. Would you mind talking a little bit about what leads you to this belief?
[linkpost] Why Going to the Doctor Sucks (WaitButWhy)

If you're interested in this, it might also be worth checking out One Medical (which I have a membership at, here's my referral code which I think only helps you?), which has the feature that appointments are easy to get soon and also the feature that the spaces are nice, while not having changed the experience of talking with the doctor all that much (which seems to be a big part of their dream, and may or may not be part of what they're delivering). In particular, One Medical doesn't the 'three-person-team' model which I imagine will be doing most of the... (read more)

Yudkowsky and Christiano discuss "Takeoff Speeds"

There have been numerous terrorist incidents in world history, and triggers to war, and it's not clear to me that 9/11 is the most visceral.

I do think part of the problem here is 'reference class tennis', where you can draw boundaries in different ways to get different conclusions, and it's not quite clear which boundaries are the most predictive.

As I understand Eliezer's point in that section, Paul's model seems to predict there won't be discontinuities in the input/output response, but we have lots of examples of that sort of thing. Two years before the ... (read more)

3Matthew Barnett5dGood point. I think I had overstated the extent to which terrorism had been a frequent occurance. 9/11 is indeed the deadliest terrorist attack ever recorded [https://en.wikipedia.org/wiki/List_of_battles_and_other_violent_events_by_death_toll#Non-state_terrorist_attacks] (I didn't realize that few other attacks even came close). However, I do want to push back against the idea that this event was totally unprecedented. The comparison to other "terrorist attacks" is, as you hint at, a bit of a game of reference class tennis. When compared to other battles, air raids, and massacres, Wikipedia notes that there have been several dozen that compare in the context of war. But of course, the United States did not see itself in an active state of war at the time. The closest comparison is probably the attack on Pearl Harbor, in which a comparable number of people died. But that attack was orchestrated by an industrializing state, not an insurgent terrorist group.
Ngo and Yudkowsky on AI capability gains

The mental move I'm doing for each of these examples is not imagining universes where addition/evolution/other deep theory is wrong, but imagining phenomena/problems where addition/evolution/other deep theory is not adapted. If you're describing something that doesn't commute, addition might be a deep theory, but it's not useful for what you want. 

Yeah, this seems reasonable to me. I think "how could you tell that theory is relevant to this domain?" seems like a reasonable question in a way that "what predictions does that theory make?" seems like it's somehow coming at things from the wrong angle.

Ngo and Yudkowsky on AI capability gains

And even if I feel what you're gesturing at, this sounds/looks like you're saying "even if my prediction is false, that doesn't mean that my theory would be invalidated". 

So, thermodynamics also feels like a deep fundamental theory to me, and one of the predictions it makes is "you can't make an engine more efficient than a Carnot engine." Suppose someone exhibits an engine that appears to be more efficient than a Carnot engine; my response is not going to be "oh, thermodynamics is wrong", and instead it's going to be "oh, this engine is making use of... (read more)

8Adele Lopez8dThat's not what it predicts. It predicts you can't make a heat engine more efficient than a Carnot engine.
2adamShimi8dThanks for the thoughtful answer! My gut reaction here is that "you can't make an engine more efficient than a Carnot engine" is not the right kind of prediction to try to break thermodynamics, because even if you could break it in principle, staying at that level without going into the detailed mechanisms of thermodynamics will only make you try the same thing as everyone else does. Do you think that's an adequate response to your point, or am I missing what you're trying to say? The mental move I'm doing for each of these examples is not imagining universes where addition/evolution/other deep theory is wrong, but imagining phenomena/problems where addition/evolution/other deep theory is not adapted. If you're describing something that doesn't commute, addition might be a deep theory, but it's not useful for what you want. Similarly, you could argue that given how we're building AIs and trying to build AGI, evolution is not the deep theory that you want to use. It sounds to me like you (and your internal-Yudkowsky) are using "deep fundamental theory" to mean "powerful abstraction that is useful in a lot of domains". Which addition and evolution fundamentally are. But claiming that the abstraction is useful in some new domain requires some justification IMO. And even if you think the burden of proof is on the critics, the difficulty of formulating the generators makes that really hard. Once again, do you think that answers your point adequately?
Ngo and Yudkowsky on AI capability gains

It's taking a massive massive failure and trying to find exactly the right abstract gloss to put on it that makes it sound like exactly the right perfect thing will be done next time.

I feel like Ngo didn't really respond to this?

Like, later he says: 

Right, I'm not endorsing this as my mainline prediction about what happens. Mainly what I'm doing here is highlighting that your view seems like one which cherrypicks pessimistic interpretations.

But... Richard, are you endorsing it as 'at all in line with the evidence?' Like, when I imagine living in that ... (read more)

8Richard_Ngo8dI think we live in a world where there are very strong forces opposed to technological progress, which actively impede a lot of impactful work, including technologies which have the potential to be very economically and strategically important (e.g. nuclear power, vaccines, genetic engineering, geoengineering). This observation doesn't lead me to a strong prediction that all such technologies will be banned; nor even that the most costly technologies will be banned - if the forces opposed to technological progress were even approximately rational, then gain of function research would be one of their main priorities (although I note that they did manage to ban it, the ban just didn't stick). But when Eliezer points to covid as an example of generalised government failure, and I point to covid as also being an example of the specific phenomenon of people being very wary of new technology, I don't think that my gloss is clearly absurd. I'm open to arguments that say that serious opposition to AI progress won't be an important factor in how the future plays out; and I'm also open to arguments that covid doesn't provide much evidence that there will be serious opposition to AI progress. But I do think that those arguments need to be made.
Ngo and Yudkowsky on alignment difficulty

I'm not sure why we haven't tried the 'generate and publish chatroom logs' option before.

My guess is that a lot of these conversations often hinge on details that people are somewhat ansy about saying in public, and I suspect MIRI now thinks the value of "credible public pessimism" is larger than the cost of "gesturing towards things that seem powerful" on the margin such that chatlogs like this are a better idea than they would have seemed to the MIRI of 4 years ago. [Or maybe it was just "no one thought to try, because we had access to in-person conversations and those seemed much better, despite not generating transcripts."]

Ngo and Yudkowsky on alignment difficulty

I'm still unsure how true I think this is.

Clearly a full Butlerian jihad (where all of the computers are destroyed) suspends AGI development indefinitely, and destroying no computers doesn't slow it down at all. There's a curve then where the more computers you destroy, the more you both 1) slow down AGI development and 2) disrupt the economy (since people were using those to keep their supply chains going, organize the economy, do lots of useful work, play video games, etc.).

But even if you melt all the GPUs, I think you have two obstacles:

  1. CPUs alone can
... (read more)
6Eliezer Yudkowsky12dI agree you might need to make additional moves to keep the table flipped, but in a scenario like this you would actually have the capability to make those moves.
Adele Lopez's Shortform

So we would need to figure out how to robustly get an honest signal from such an experiment, which still seems quite hard. But perhaps it's easier than solving the full alignment problem before the first shot.

IMO this is a 'additional line of defense' boxing strategy instead of simplification. 

Note that in the traditional version, the 'dud' bit of the bomb can only be the trigger; a bomb that absorbs the photon but then explodes isn't distinguishable from a bomb that absorbs the photon and then doesn't explode (because of an error deeper in the bomb).... (read more)

1Adele Lopez12dThanks!
A Critique of Functional Decision Theory

There's a period at the end of the URL that was automatically included; deleting that fixes the issue (I've edited your comment accordingly).

1Heighn14dHey, thanks! That's awesome.
Discussion with Eliezer Yudkowsky on AGI interventions

I'm not sure why you mean by 'philosophically' simple? 

I think if we had the right conception of goals, the difference between 'corrigibility' and 'incorrigibility' would be a short sentence in that language. (For example, if you have a causal graph that goes from "the state of the world" to "my observations", you specify what you want in terms of the link between the state of the world and your observations, instead of the observations.)

This is in contrast to, like, 'practically simple', where you've programmed in rules to not do any of the ten thousand things it could do to corrupt things.

Speaking of Stag Hunts

Huh, not sure how I missed that; thanks for pointing it out.

Discussion with Eliezer Yudkowsky on AGI interventions

However, I think it's not at all obvious to me that corrigibility doesn't have a "small central core". It does seem to me like the "you are incomplete, you will never be complete" angle captures a lot of what we mean by corrigibility. 

I think all three of Eliezer, you, and I share the sense that corrigibility is perhaps philosophically simple. The problem is that for it to actually have a small central core / be a natural stance, you need the 'import philosophy' bit to also have a small central core / be natural, and I think those bits aren't true.

Lik... (read more)

1LawChan16dI'm not sure why you mean by 'philosophically' simple? Do you agree that other problems in AI Alignment don't have "philosophically' simple cores? It seems to me that, say, scaling human supervision to a powerful AI or getting an AI that's robust to 'turning up the dial' seem much harder and intractable problems than corrigibility.
Discussion with Eliezer Yudkowsky on AGI interventions

Oh, I was imagining something like "well, our current metals aren't strong enough, what if we developed stronger ones?", and then focusing on metallurgy. And this is making forward progress--you can build a taller tower out of steel than out of iron--but it's missing more fundamental issues like "you're not going to be able to drive on a bridge that's perpendicular to gravity, and the direction of gravity will change over the course of the trip" or "the moon moves relative to the earth, such that your bridge won't be able to be one object", which will sink... (read more)

Discussion with Eliezer Yudkowsky on AGI interventions

Certainly, if you're working on a substantial breakthrough in AI capability, there are reasons to keep it secret. But why would you work on that in the first place?

Most of the mentions of secrecy in this post are in that context. I think a lot of people who say they care about the alignment problem think that the 'two progress bars' model, where you can work on alignment and capability independent of each other, is not correct, and so they don't see all that much of a difference between capability work and alignment work. (If you're trying to predict human... (read more)

8Vanessa Kosoy16dIf there's no difference between capability work and alignment work, then how is it possible to influence anything at all? If capability and alignment go hand in hand, then either transformative capability corresponds to sufficient alignment (in which case there is no technical problem) or it doesn't (in which case we're doomed). The only world in which secrecy makes sense, AFAICT, is if you're going to solve alignment and capability all by yourself. I am extremely skeptical of this approach.
Discussion with Eliezer Yudkowsky on AGI interventions

I'm annoyed by EY (and maybe MIRI's?) dismissal of every other alignment work, and how seriously it seems to be taken here, given their track record of choosing research agendas with very indirect impact on alignment

For what it's worth, my sense is that EY's track record is best in 1) identifying problems and 2) understanding the structure of the alignment problem.

And, like, I think it is possible that you end up in situations where the people who understand the situation best end up the most pessimistic about it. If you're trying to build a bridge to the ... (read more)

6adamShimi16dAgreed on the track record, which is part of why that's so frustrating he doesn't give more details and feedback on why all these approaches are doomed in his view. That being said, I disagree for the second part, probably because we don't mean the same thing by "moving the ball"? In your bridge example, "moving the ball" looks to me like trying to see what problems the current proposal could have, how you could check them, what would be your unknown unknowns. And I definitely expect such an approach to find the problems you mention. Maybe you could give me a better model of what you mean by "moving the ball"?
Speaking of Stag Hunts

Where? Is it the quoted lines?

4Duncan_Sabien16d
Speaking of Stag Hunts

Perhaps you are suggesting a post that does that-and-nothing-but-that?

I think I am suggesting "link to things when you mention them." Like, if I want to argue with DanielFilan about whether or not a particular garment "is proper" or not, it's really not obvious what I mean, whereas if I say "hey I don't think that complies with the US Flag Code", most of the work is done (and then we figure out whether or not section j actually applies to the garment in question, ultimately concluding that it does not).

Like, elsewhere you write:

The standard is: don't viola

... (read more)
2Duncan_Sabien17dSabien's Sins is linked in the OP (near the end, in the list of terrible ideas). I will probably make a master linkpost somewhere in my next four LW essays. Thanks.
2Said Achmiz17dIndeed, my opinion of “double crux” has not improved since the linked comments were written.
Speaking of Stag Hunts

The first ones that come to my mind are "money" and "being published in physical books".

So I think the Review is pretty good at getting good old content, but I think the thing Said is talking about should happen more quickly, and should be more like Royal Society Letters or w/e.

Actually, I wonder about Rohin's newsletters as a model/seed. They attract more scrutiny to things, but they come with the reward of Rohin's summary (and, presumably, more eyeballs than it would have gotten on its own). But also people were going to be writing those things for their... (read more)

Speaking of Stag Hunts

because the comments are exhausting and costly in approximately the ways I'm gesturing at.

(We have some disagreement but more overlap than disagreement.)

As I understand Ben Pace, he's saying something like "I want people to take more risks so that we find more gold", and you're replying with something like "I think people will take more risks if we make the space more safe, by policing things like strawmanning."

It seems central to me to somehow get precise and connected to reality, like what specific rules you're suggesting policing (strawmanning? projecti... (read more)

1Duncan_Sabien17dI note that I've already put something like ten full hours into creating exactly these types of examples, and that fact sort of keeps getting ignored/people largely never engage with them. Perhaps you are suggesting a post that does that-and-nothing-but-that?
7dxu17dStrong-upvote as well for the specificity request; the place where I most strongly expect attempts at "increasing standards" to fail is the point where people realize that broad agreement about direction does not necessarily translate to finer agreement about implementation, and I expect this is best avoided by sharing gears-level models as quickly and as early during the initial discussion as possible. As I wrote in another comment [https://www.lesswrong.com/posts/D5BP9CxKHkcjA7gLv/speaking-of-stag-hunts?commentId=LiqzmQRPhCSmaqog6] :
Speaking of Stag Hunts

Huh, I see the post plus a big "log in" bar at the bottom on Safari 15612.1.29.41.4 (Mac), and the same without the bar in an incognito tab Chrome 94.0.4606.71 (Mac). These don't overlap with any of the things you tried, but it's strange to me that our results are consistently different.

Vaniver's Shortform

I was hiking with housemates yesterday, and we chanced across the San Francisco Discovery Bay site. Someone erected a monument to commemorate the Portala Expedition; apparently about a year ago someone else defaced the monument to remove the year, name, and the phrase "discovered".

Which made me wonder: what would a more neutral name be? Clearly they did something, even tho there were already humans living in the area. A housemate suggested "were surprised by" as a replacement for discovered, and I found it amusing how well it fit. (Especially other cases, ... (read more)

2Dagon18dPerhaps "reported to Europe" as a description? I think it's different from a child's learning, or about the internal state of the sailors' beliefs. Though honestly, I don't object to "discovered" - it's a common enough usage that I don't think there's any actual ambiguity.
Ruling Out Everything Else

I don't think that if you don't believe the first speaker to be well-intentioned, you should pretend like they are.  But even in that case, there are better forward moves than "Bullshit," especially on LW.

(For instance, you could go the route of "if [cared], would [do X, Y, Z].  Since ¬Z, can make a marginal update against [cared].")

Somehow this reminds me of a recent interaction between ESRogs and Zack_M_Davis, where I saw ESRogs as pushing against slurs in general, which I further inferred to be part of pushing against hostility in general. But... (read more)

2Duncan_Sabien20dI ended up jumping in to bridge the inferential gap in that exact exchange. =P I think it's fine to discover/decide that your [enemy] rating should go up, and I still don't think that means "abandon the principles." (EDIT: to be clear, I don't think you were advocating for that.) It might mean abandon some of the disarmaments that are only valid for other peace-treaty signatories, but I don't think there are enmities on LW in which it's a good idea to go full Dark Arts, and I think it would be good if the mass of users downvoted even Eliezer if he were doing so in the heat of the moment.
Speaking of Stag Hunts

I think I agree with the models here, and also want to add a complicating factor that I think impacts the relevance of this.

I think running a site like this in a fully consequentialist way is bad. When you're public and a seed of something, you want to have an easily-understandable interface with the world; you want it to be the case that other people who reason about you (of which there will be many, and who are crucial to your plan's success!) can easily reason about you. Something more like deontology or virtue ethics ("these are the rules I will follow... (read more)

+1 to all this, and in particular I'm very strongly on board with rationality going beyond AI safety. I'm a big fan of LessWrong's current nominal mission to "accelerate intellectual progress", and when I'm thinking about making progress in a high-dimensional world, that's usually the kind of progress I'm thinking about. (... Which, in turn, is largely because intellectual/scientific/engineering progress seem to be the "directions" which matter most for everything else.)

Speaking of Stag Hunts

And maybe I should update in that direction, and just ignore a constant background shrieking.

I'm not sure about this; there is some value in teaching undergrads rigor, and you seem more motivated to than I am. And, like, I did like Logan's comment about rumor, and I think more people observing things like that sooner is better. I think my main hope with the grandparent was to check if you're thinking the rigor is the moon or the finger, or something.

9Duncan_Sabien21dMy views here aren't fully clarified, but I'm more saying "the pendulum needs to swing this way for LessWrong to be good" than saying "LessWrong being good is the pendulum being all the way over there." Or, to the extent that I understood you and am accurately representing Ben Pace, I agree with you both.
Speaking of Stag Hunts

EDIT: also last time you made a comment on one of my posts and I answered back, I never heard back from you and I was a little Sad so could you at least leave me a "seen" or something

Seen, also, which are you thinking of? I might have had nothing to say, or I might have just been busy when I saw the response and I wasn't tracking that I should respond to it.

2Duncan_Sabien21dhttps://www.lesswrong.com/posts/57sq9qA3wurjres4K/ruling-out-everything-else?commentId=fdapBc7dnKEhJ4no4
Speaking of Stag Hunts

I think this is a fair summary of what we said years ago. I'm not sure how much people's minds have changed on the issue. I think Ben Pace's warning of Said (you have to scroll to the bottom of the page and then expand that thread) and the related comments are probably the place to look, including habryka's comment here.

Before checking the moderation list, the posts that come to mind (as a place to start looking for this sort of conversation) were Kensho (where I think a lot of the mods viewed Said as asking the right questions) and Meta-Discussion from Ci... (read more)

Speaking of Stag Hunts

I'll have more to say on this in the future, but for now I just want to ramble about something.

I've been reading through some of the early General Semantics works. Partially to see if there are any gaps in my understanding they can fill, partially as a historical curiosity (how much of rationality did they have figured out, and what could they do with it?), partially because it might be good fodder for posts on LW (write a thousand posts to Rome).

And somehow a thing that keeps coming into my mind while reading them is the pre-rigorous/rigorous/post-rigorou... (read more)

4Said Achmiz20dThis is an instance of three levels of mastery [https://blog.obormot.net/Three-levels-of-mastery].
6Ben Pace21dThis comment is surprising to me in how important I think this point is.

For me, what this resonates most clearly with is the interaction I just had with Ben Pace.

Ben was like "X"

And I was like "mostly yes to X, but also no to the implicature that I think surrounds X which is pretty bad."

And Ben was like "oh, definitely not that!  Heck no!  Thanks for pointing it out, but no, and also I think I'll change nothing in response to finding out that many people might draw that from X!"

And my response was basically "yeah, I don't think that Ben Pace on the ideal LessWrong should do anything different."

Because the thing Ben s... (read more)

[Book Review] "The Bell Curve" by Charles Murray

What do you mean, “would’ve”? 

I understand Ape in the coat to be saying the bit from I Can Tolerate Anything Except the Outgroup, “Thus do you gain no merit by tolerating them!”, implying that you have nothing against fascists, and contrasting that to book review by a woke leftist, like perhaps the White Fragility review that was posted back in September.

3Said Achmiz22dIf so, then that’s an absurd thing to say. Given my background, saying that I have nothing against fascists is one heck of a claim…
[Book Review] "The Bell Curve" by Charles Murray

AI alignment was never political remotely like how the Bell Curve is political.

Not within the mainstream politics, but within academic / corporate CS and AI departments. 

2steven046122dYou'd have to use a broad sense of "political" to make this true (maybe amounting to "controversial"). Nobody is advocating blanket avoidance of controversial opinions, only blanket avoidance of narrow-sense politics, and even then with a strong exception of "if you can make a case that it's genuinely important to the fate of humanity in the way that AI alignment is important to the fate of humanity, go ahead". At no point could anyone have used the proposed norms to prevent discussion of AI alignment.
[Book Review] "The Bell Curve" by Charles Murray

Our culture is desperately in need of spaces that are correct about the most important technical issues

I also care a lot about this; I think there are three important things to track.

First is that people might have reputations to protect or purity to maintain, and so want to be careful about what they associate with. (This is one of the reasons behind the separate Alignment Forum URL; users who wouldn't want to post something to Less Wrong can post someplace classier.)

Second is that people might not be willing to pay costs to follow taboos. The more a spac... (read more)

4steven046123dSome more points I want to make: * I don't care about moderation decisions for this particular post, I'm just dismayed by how eager LessWrongers seem to be to rationalize shooting themselves in the foot, which is also my foot and humanity's foot, for the short term satisfaction of getting to think of themselves as aligned with the forces of truth in a falsely constructed dichotomy against the forces of falsehood. * On any sufficiently controversial subject, responsible members of groups with vulnerable reputations will censor themselves if they have sufficiently unpopular views, which makes discussions on sufficiently controversial subjects within such groups a sham. The rationalist community should oppose shams instead of encouraging them. * Whether political pressure leaks into technical subjects mostly depends on people's meta-level recognition that inferences subject to political pressure are unreliable, and hosting sham discussions makes this recognition harder. * The rationalist community should avoid causing people to think irrationally, and a very frequent type of irrational thinking (even among otherwise very smart people) is "this is on the same website as something offensive, so I'm not going to listen to it". "Let's keep putting important things on the same website as unimportant and offensive things until they learn" is not a strategy that I expect to work here. * It would be really nice to be able to stand up to left wing political entryism, and the only principled way to do this is to be very conscientious about standing up to right wing political entryism, where in this case "right wing" means any politics sufficiently offensive to the left wing, regardless of whether it thinks of itself as right wing. I'm not as confident about these conclusions as it sounds, but my lack of confidence comes from seeing that people whose judgment I trust disagree, and it does not come from the argumen
5steven046123dI agree that LW shouldn't be a zero-risk space, that some people will always hate us, and that this is unavoidable and only finitely bad. I'm not persuaded by reasons 2 and 3 from your comment at all in the particular case of whether people should talk about Murray. A norm of "don't bring up highly inflammatory topics unless they're crucial to the site's core interests" wouldn't stop Hanson from posting about ems, or grabby aliens, or farmers and foragers, or construal level theory, or Aumann's theorem, and anyway, having him post on his own blog works fine. AI alignment was never political remotely like how the Bell Curve is political. (I guess some conceptual precursors came from libertarian email lists in the 90s?) If AI alignment becomes very political (e.g. because people talk about it side by side with Bell Curve reviews), we can invoke the "crucial to the site's core interests" thing and keep discussing it anyway, ideally taking some care to avoid making people be stupid about it. If someone wants to argue that having Bell Curve discussion on r/TheMotte instead of here would cause us to lose out on something similarly important, I'm open to hearing it.
[Book Review] "The Bell Curve" by Charles Murray

Elsewhere you write (and also ask to consolidate, so I'm responding here):

The main disagreement seems to come down to how much we would give up when disallowing posts like this. My gears model still says 'almost nothing' since all it would take is to extend the norm "let's not talk about politics" to "let's not talk about politics and extremely sensitive social-justice adjacent issues", and I feel like that would extend the set of interesting taboo topics by something like 10%.

I think I used to endorse a model like this much more than I do now. A particula... (read more)

This is a helpful addendum. I didn't want to bust out the slippery slope argument because I didn't have clarity on the gears-level mechanism. But in this case, we seem to have a ratchet in which X is deemed newly offensive, and a lot of attention is focused on just this particular word or phrase X. Because "it's just this one word," resisting the offensive-ization is made to seem petty - wouldn't it be such a small thing to give up, in exchange for inflicting a whole lot less suffering on others?

Next week it'll be some other X though, and the only way this... (read more)

2steven046124dI don't think it will be obvious, but I think we'll be able to make an imperfect estimate of when to change the policy that's still better than giving up on future evaluation of such tradeoffs and committing reputational murder-suicide immediately. (I for one like free speech and will be happy to advocate for it on LW when conditions change enough to make it seem anything other than pointlessly self-destructive.)
2020 PhilPapers Survey Results

Virtue ethicists one-box, because (a subset of) one-boxers are the ones talking about 'making yourself into the right kind of agent'.

This seems sort of obvious to me, and I'm kind of surprised that only a bit over half of the virtue ethicists one-box.

[EDIT] I think I was giving the virtue ethicists too much credit and shouldn't have been that surprised--this is actually a challenging situation to map onto traditional virtues, and 'invent the new virtue for this situation' is not that much a standard piece of virtue ethics. I would be surprised if only a bit over half of virtue ethicists pay up in Parfit's Hitchhiker, even tho the problems are pretty equivalent.

[Book Review] "The Bell Curve" by Charles Murray

In ten or twenty or forty years from now, in a way that's impossible to predict because any specific scenario is extremely unlikely, the position to be worried about AGI will get coupled to being anti social justice in the public discourse, as a result it will massively lose status and the big labs react by taking safety far less seriously and maybe we have fewer people writing papers on alignment

So, I both think that in the past 1) people have thought the x-risk folks are weird and low-status and didn't want to be affiliated with them, and in the present ... (read more)

3Rafael Harth24dI mean, this works until someone in a position of influence bows the the pressure, and I don't see why this can't happen. The main disagreement seems to come down to how much we would give up when disallowing posts like this. My gears model still says 'almost nothing' since all it would take is to extend the norm "let's not talk about politics" to "let's not talk about politics and extremely sensitive social-justice adjacent issues", and I feel like that would extend the set of interesting taboo topics by something like 10%. (I've said the same here [https://www.lesswrong.com/posts/vvc2MiZvWgMFaSbhx/book-review-the-bell-curve-by-charles-murray?commentId=boLrgyD3pof7Dztub] ; if you have a response to this, it might make sense to all keep it in one place.)
Ruling Out Everything Else

The second speaker ... well, the second speaker probably did say exactly what they meant, connotation and implication and all.  But if I imagine a better version of the second speaker, one who is trying to express the steelman of their point and not the strawman, it would go something like:

"Okay, so, I understand that you're probably just trying to help, and that you genuinely want to hear people's stories so that you can get to work on making things better.  But like.  You get how this sounds, right?  You get how, if I'm someone who's

... (read more)
2Duncan_Sabien1moI don't think I'm advocating steelmanning for cooperativeness and morality; that was a clumsy attempt to gesture in the direction of what I was thinking. EDIT: Have made an edit to the OP to repair this Or, to put it another way, I'm not saying "starting from the second speaker's actual beliefs, one ought to steelman." Rather, I simply do not hold the second speaker's actual beliefs; I was only able to engage with their point at all by starting with my different set of predictions/anticipations, and then presenting their point from that frame. Like, even if one believes the first speaker to be well-intentioned, one can nevertheless sympathize with, and offer up a cooperative version of, the second speaker's objection. I don't think that if you don't believe the first speaker to be well-intentioned, you should pretend like they are. But even in that case, there are better forward moves than "Bullshit," especially on LW. (For instance, you could go the route of "if [cared], would [do X, Y, Z]. Since ¬Z, can make a marginal update against [cared].")
Zoe Curzi's Experience with Leverage Research

I'm sort of surprised that you'd interpret that as a mistake. It seems to me like Eliezer is running a probabilistic strategy, which has both type I and type II errors, and so a 'mistake' is something like "setting the level wrong to get a bad balance of errors" instead of "the strategy encountered an error in this instance." But also I don't have the sense that Eliezer was making an error.

1TekhneMakre1moIt sounds like this describes every strategy? I guess you mean, he's explicitly taking into account that he'll make errors, and playing the probabilities to get good expected value. So this makes sense, like I'm not saying he was making a strategic mistake by not, say, working with Geoff. I'm saying: sounds like he's conflating shareable and non-shareable evidence. Geoff could have seen a bunch of stuff and learned heuristics that he couldn't articulately express other than with silly-seeming "bright-line psychoanalytic rules written out in English". Again, it can make sense to treat this as "for my purposes, equivalent to being obviously wrong". But like, it's not really equivalent, you just *don't know* whether the person has hidden evidence.
Zoe Curzi's Experience with Leverage Research

Geoff describes being harmed by some sort of initial rejection by the rationality/EA community (around 2011? 2010?).

One of the interesting things about that timeframe is that a lot of the stuff is online; here's the 2012 discussion (Jan 9th, Jan 10th, Sep 19th), for example. (I tried to find his earliest comment that I remembered, but I don't think it was with the Geoff_Anders account or it wasn't on LessWrong; I think it was before Leverage got started, and people responded pretty skeptically then also?)

6TekhneMakre1moThanks! One takeaway: Eliezer's interaction with Geoff [https://www.lesswrong.com/posts/8j4zirwfhWhT8nwsc/a-critique-of-leverage-research-s-connection-theory?commentId=uZ5i7PMEn2uGMhCKc] does seem like Eliezer was making some sort of mistake. Not sure what the core is, but, one part is like conflating [evidence, the kind that can be interpersonally verified] with [evidence, the kind that accumulates subconsciously as many abstract percepts and heuristics, which can be observably useful while still pre-theoretic, pre-legible]. Like, maybe Eliezer wants to only talk with people where either (1) they already have enough conceptual overlap that abstract cutting-edge theories also always cash out as perceivable predictions, or (2) aren't trying to share pre-legible theories. But that's different from Geoff making some terrible incurable mistake of reasoning. (Terrible incurable mistakes are certainly correlated with illegibility, but that's not something to Goodhart.)
My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

I think most of LW believes we should not risk ostracizing a group (with respect to the rest of the world) that might save the world, by publicizing a few broken eggs. If that's the case, much discussion is completely moot. I personally kinda think that the world's best shot is the one where MIRI/CFAR type orgs don't break so many eggs. And I think transparency is the only realistic mechanism for course correction. 

FWIW, I (former MIRI employee and current LW admin) saw a draft of this post before it was published, and told jessicata that I thought she should publish it, roughly because of that belief in transparency / ethical treatment of people.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

they randomly make big errors

I think it's important that the errors are not random; I think you mean something more like "they make large opaque errors."

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

Were you criticized for socializing with people outside MIRI/CFAR, especially with "rival groups"?

As a datapoint, while working at MIRI I started dating someone working at OpenAI, and never felt any pressure from MIRI people to drop the relationship (and he was welcomed at the MIRI events that we did, and so on), despite Eliezer's tweets discussed here being a pretty widespread belief at MIRI. (He wasn't one of the founders, and I think people at MIRI saw a clear difference between "founding OpenAI" and "working at OpenAI given that it was founded", so idk... (read more)

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

I believe Anthropic doesn't expect its employees to be in the office every day, but I think this is more pandemic-related than it is a deliberate organizational design choice; my guess is that most Anthropic employees will be in the office a year from now.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

On the other side of it, why do people seem TOO DETERMINED to turn him into a scapegoat? Most of you don't sound like you really know him at all.

A blogger I read sometimes talks about his experience with lung cancer (decades ago), where people would ask his wife "so, he smoked, right?" and his wife would say "nope" and then they would look unsettled. He attributed it to something like "people want to feel like all health issues are deserved, and so their being good / in control will protect them." A world where people sometimes get lung cancer without havi... (read more)

+1 to your example and esp "isn't owning the degree to which the effects they're worried about are caused by their instability or the them-Michael dynamic." 

I also want to leave open the hypothesis that this thing isn't a one-sided dynamic, and Michael and/or his group is unintentionally contributing to it. Whereas the lung cancer example seems almost entirely one-sided. 

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

Somehow this reminds me of the time I did a Tarot reading for someone, whose only previous experience had been Brent Dill doing a Tarot reading, and they were... sort of shocked at the difference. (I prefer three card layouts with a simple context where both people think carefully about what each of the cards could mean; I've never seen his, but the impression I got was way more showmanship.)

If it works as a device to facilitate sub-conscious associations, then maybe an alternative should be designed that sheds the mystical baggage and comes with clear explanations of why and how it works. 

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

Note that there's an important distinction between "corporate management" and "corporate employment"--the thing where you say "yeesh, I'm glad I'm not a manager at Google" is substantially different from the thing where you say "yeesh, I'm glad I'm not a programmer at Google", and the audience here has many more programmers than managers.

[And also Vanessa's experience matches my impressions, tho I've spent less time in industry.]

[EDIT: I also thought it was clear that you meant this more as a "this is what MIRI was like" than "MIRI was unusually bad", but ... (read more)

My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.

Google's a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.

Load More