All of Ben Pace's Comments + Replies

For the record, our relationship to supporting events for this ecosystem is changing from something like "all of our resources are the same, here have my venue for free if you need it" to "markets and pricing are a great way for large masses of people to coordinate on the value of a good or service, let's coordinate substantially via trade". 

For instance, during a previous cohort of SERI MATS scholars at the Lightcone Offices, I spent a couple of weeks of work adding a second floor and getting it furnished and doing interior design, hiring another sup... (read more)

Here are some things I like about owning this space:

  • You don't have to ask anyone's permission to make modifications to it or to use it in unusual ways. For instance, if we want to cover every wall with whiteboards (as we've done in many rooms), we don't have to ask anyone's permission, we just order the boards, hire a few laborers, and have them cut them to size and nail them all along the walls. If I want to turn a 4-person team office into a 4-person bedroom, I just get a few people, carry the furniture into storage, carry some beds in, and we're done. T
... (read more)
5gwern1d
(I'd enjoy hearing more about the background/history of that temple.)
3rachelAF3d
These all sound like major benefits to owning the venue yourself! To be clear, I don't doubt at all that using the Inn for events is much better than non-purpose-built space. However, the Inn also has costs that renting existing spaces wouldn't: I assume that purchasing and renovating it costs more than renting hotel spaces as-needed for events (though please correct me if I'm wrong!), and my impression is that it's taken the Lightcone team a lot of time and effort over the past year+ to purchase and renovate, which naturally has opportunity costs. I'm asking because my uninformed guess is that those financial and time costs outweigh the (very real) benefits of hosting events like you have been. I'm interested to hear if I'm just wrong about the costs, or if you have additional plans to make even more effective use of the space in the future, or if there's additional context I'm missing. ETA: Oli answered these questions below, so no need to respond to them unless you have something additional you'd like me to know.

Thanks, I was confused that I couldn't find it.

  1. He cites ARC’s GPT-4 evaluation and Lesswrong in his AI report which has a large section on safety.

I wanted to double-check this. 

The relevant section starts on page 94, "Section 4: Safety", and those pages cite in their sources around 10-15 LW posts for their technical research or overviews of the field and funding in the field. (Make sure to drag up the sources section to view all the links.)

Throughout the presentation and news articles he also has a few other links to interviews with ppl on LW (Shane Legg, Sam Altman, Katja Grace).

3Hauke Hillebrandt4d
ARC's GPT-4 evaluation is cited in the FT article [https://archive.ph/V6r2j#selection-2309.0-2319.296], in case that was ambiguous.

Thanks! This is fairly tempting. I'm a bit concerned by 

Some other explanation that's of this level of "very weird"

To be clear, if it were just the 4 hypotheses you mention, then I feel pretty good about this, and I'd just want to reflect over 200:1 versus 100:1. 

  • For instance, if some form of undiscovered bat or sea creature is actually good at flying and that explains the images, does that count as "very weird"? That is, it's not an animal that can design technologies, but it is the cause of a bunch of the UFO reportings.
  • Also "psychic phenomenon
... (read more)
1RatsWrongAboutUAP4d
I have created a post for this bet https://www.lesswrong.com/posts/t5W87hQF5gKyTofQB/ufo-betting-put-up-or-shut-up
1RatsWrongAboutUAP4d
If a skeptic/debunker would feel smug and vindicated about any given explanation, then that explanation would NOT resolve in my favor. Weird animal (As mentioned, barring that they aren't secretly on par with humans), is prosaic and would NOT resolve in my favor.   Merely advanced "mentalist" type stuff would NOT count. This class is more pointing at "true" psychic phenomena. ie, something more like ESP, remote viewing, astral projection, etc.   I lurk, but not closely enough to know what handles would be good for this bet. Ill accept this list, and also throw in Scott Alexander and Eliezer.

I have not read this post, and I have not looked into whatever the report is, but I'm willing to take a 100:1 bet that there is no such non-human originating craft (by which I mean anything actively designed by a technological species — I do not mean that no simple biological matter of any kind could not have arrived on this planet via some natural process like an asteroid), operationalized to there being no Metaculus community forecast (or Manifold market with a sensible operationalization and reasonable number of players) that assigns over 50% probabilit... (read more)

1RatsWrongAboutUAP5d
Willing to bet on this at 200:1? (This is what I have been offering others) My Offer: * I Send you $X Immediately, You pay out $200*X if I win * I want a 5 year time horizon. * I am betting that one of the "very weird" hypotheses turns out true. Bet resolves in my favor if within 5 years its determined that >0 UAP are explained by: * Literal aliens * Magic/spiritual/paranormal/psychic phenomenon * Time travel * Leftovers of an ancient civilization / Some other unknown non-human advanced civilization on earth * Some other explanation that's of this level of "very weird" * Explicitly excluding merely hyper-advanced human tech * I forfeit any and all potential "gotcha" cases. * Determination of resolution to be up to you. * I reserve the right to appeal to the LW community. [I will not abuse this right] Please let me know what your maximum payout you swear to be good for would be, and Ill send you 1/200th that today, up to a maximum of 5K. If these terms are acceptable, please provide a means for me to pay you. I would prefer a crypto address, but will make whatever work. Upon payment please post an acknowledgement of payment. I offer the same terms to anyone else [with a nontrivial post history]. 

"AI maniacs" is maybe a term that meets this goal? Mania is the opposite side to depression, both of which are about having false beliefs just in opposite emotionally valenced directions, and also I do think just letting AI systems loose in the economy is the sort of thing a maniac in charge of a civilization would do.

The rest of my quick babble: "AI believers" "AI devotee" "AI fanatic" "AI true believer" "AI prophets" "AI ideologue" "AI apologist" "AI dogmatist" "AI propagandists" "AI priests".

I had the first of these pieces printed and left on my desk to read the other day.

3Maxwell Tabarrok14d
That's awesome to hear! Hope you enjoy

I think I tend to base my level of alarm on the log of the severity*probability, not the absolute value. Most of the work is getting enough info to raise a problem to my attention to be worth solving. "Oh no, my house has a decent >30% chance of flooding this week, better do something about it, and I'll likely enact some preventative measures whether it's 30% or 80%." The amount of work I'm going to put into solving it is not twice as much if my odds double, mostly there's a threshold around whether it's worth dealing with or not.

Setting that aside, it ... (read more)

2Wei Dai13d
Yeah, I think this is a factor. Paul talked a lot about "1/trillion kindness" as the reason for non-extinction, but 1/trillion kindness seems to directly imply a small utopia where existing humans get to live out long and happy lives (even better/longer lives than without AI) so it seemed to me like he was (maybe unintentionally) giving the reader a frame of “50% extinction, 50% small utopia”, while still writing other things under the “50% extinction, 50% not-extinction” frame himself.

(Strong-upvote, weak-disagree. I sadly don't have time right now to reflect and write why I disagree with this position but I hope someone else who disagrees does.)

Relatedly, when we made DontDoxScottAlexander.com, we tried not to wade into a bigger fight about the NYT and other news sites, nor to make it an endorsement of Scott and everything he's ever written/done. It just focused on the issue of not deanonymizing bloggers when revealing their identity is a threat to their careers or personal safety and there isn't a strong ethical reason to do so. I know more high-profile people signed it because the wording was conservative in this manner.

I also have a strong personal rule against making public time-bound commitments unless I need to. I generally regret it because unexpected things come up and I feel guilty about not replying in the time frame I thought I would.

I might be inclined to hit a button that says "I hope to respond further to this".

I've just had an interesting experience that changed my felt-sense of consciousness and being embodied.

I've played over 80 hours of the newly released Zelda game, which is a lot given that it's only been out for 14 days. I do not normally play video games very much, this has been a fairly drastic change in how I've spent my personal time.

I'm really focused while playing it, and feel very immersed in the world of the game. So much so that I had a quite odd experience coming back to the rest of my life.

Yesterday, after playing the game for an hour, I wandere... (read more)

I find it amusing that I can now both agree and disagree with a comment.

1Max H22d
I anti-agree with this comment. I also anti-disagree with it! 

Curated! I loved a lot of things about this post. 

I think the post is doing three things, all of which I like. First, it documents what it was like for Joe as he made substantial updates about the world. Secondly, it exhibits the rationalist practice of explaining what those updates look like using the framework of probabilities, and considering what sorts of updates a rational agent would make in his position, and contrasted that with a helpful explicit model of how a human being would make updates (e.g. using its guts). And third it's a serious and ... (read more)

You're right. Thanks!

“No,” says the philosopher. “It’s not a false dichotomy! For the sake of argument we’re suspending the laws of physics. Didn’t Galileo do the same when he banished friction from his thought experiment?” Yes, but a general rule of thumb emerges from the comparison: the utility of a thought experiment is inversely proportional to the size of its departures from reality.

This is a good point (and I think I occasionally make this mistake of giving far too unrealistic or rare counterexamples), but I do want to say that sometimes there is substantive disagreement about whether the counter-example is an extreme case or a relatively central one. 

I think of people who are willing to accept very non-central counterexamples as relevant as being very conservative on the dimension of trusting their own taste, in that they are trying to avoid using their own judgment about what counts as central. (Mostly this seems good to me in... (read more)

2Jayson_Virissimo1mo
IIRC, he says that in Intuition Pumps and Other Tools for Thinking.

It seems to me like you would like to be able to succeed at preventing an extinction-level threat without having to be competent at anything. I think reality has higher standards than that.

Occupy Wall Street is exactly the wrong example to mention, and further makes me think you have no sense of what a successful historical protest looks like.

I'm not into random people showing up and trying to command the political force of a web forum that they've made zero contributions to and trying to direct it into a poorly thought out and haphazardly-aimed effort.

I am well-aware of the stakes here, but that doesn't mean bad plans suddenly work.

These don’t seem very relevant counterarguments, I think literally all are from people who believe that AGI is an extinction-level threat soon facing our civilization.

Perhaps you mean “>50% of extinction-level bad outcomes” but I think that the relevant alternative viewpoint that would calm someone is not that the probability is only 20% or something, but is “this is not an extinction-level threat and we don’t need to be worried about it”, for which I have seen no good argument for (that engages seriously with any misalignment concerns).

2Christopher King2mo
Well, I was asking because I found Yudkowsky's model of AI doom far more complete than any other model of the long term consequences of AI. So the point of my original question is "how frequently is a model that is far more complete than it's competitors wrong?". But yeah, even something as low as 1% chance of doom demands very large amount of attentions from the human race (similar to the amount of attention we assigned to the possibility of nuclear war). (That said, I do think the specific value of p(doom) is very important when deciding which actions to take, because it effects the strategic considerations in the play to your outs [https://www.lesswrong.com/posts/xF7gBJYsy6qenmmCS/don-t-die-with-dignity-instead-play-to-your-outs] post.)

These are some of the reasons I don't want to join your protest: You've done basically zero of the hard work required to rally people behind a successful protest (other than write this announcement). You'd really need a concrete policy ask before I would think of joining your protest, and show that you're going to be able to coordinate a bunch of work from lots of people. More generally I don't really like the dynamic where the first person to say "me" is suddenly able to direct a bunch of free-energy, even if they probably aren't able to follow-through. A... (read more)

-3Alistair1mo
Isn't this how most social movements start – with a single protest, attended by a small number of people? I think this is why Percy posted here: to discuss what that might look like! And perhaps he doesn't need specific demands – look at Occupy Wall Street as an example of a movement with underspecified/vague demands that was effective in some ways (and failed in others). Again – surely this is how all social movements start? This picket won't be perfect; in my view it will highly likely be better than nothing. Do you have any suggestions?

Oops, I was unclear in my last line.

I was attempting to distinguish between someone getting angry with you and shouting at you and then punching you, and someone who is quiet and doesn’t look at you and isn’t talking much to you who then walks over and punches you.

Both are alarming. To me the latter feels more unpredictable and more alarming because I’m getting no info about when it will happen, but if someone is getting visibly angry and escalates and you don’t know where their lines are for conflict, then I can see that being more intensely alarming.

There’s nothing special about taking responsibility for something big or small. It’s the same meaning.

Within teams I’ve worked in it has meant:

  • You can be confident that someone is personally optimizing to achieve the goal
  • Both the shame of failing and the glory of succeeding will primarily accrue to them
  • There is a single point of contact for checking in about any aspect of the problem.
  • For instance, if you have an issue with how a problem is being solved, there is a single person you can go to to complain
  • Or if you want to make sure that something you’r
... (read more)

For example, as far as the “normatively correct general principles” thing goes—alright, so you think I’m factually incorrect about this particular thing I said once.[1] Let’s take for granted that I disagree. Well, and is that… a moderation-worthy offense? To disagree (with the mods? with the consensus—established how?—of Less Wrong? with anyone?) about what is essentially a philosophical claim? Are you suggesting that your correctness on this is so obvious that disagreeing can only constitute either some sort of bad faith, or blameworthy ignorance? That h

... (read more)

The claim I understand Ray to be making is that he believes you gave a false account of the site-wide norms about what users are obligated to do

Is that really the claim? I must object to it, if that’s so. I don’t think I’ve ever made any false claims about what social norms obtain on Less Wrong (and to the extent that some of my comments were interpreted that way, I was quick to clearly correct that misinterpretation).

Certainly the “normatively correct general principles” comment didn’t contain any such false claims. (And Raemon does not seem to be clai... (read more)

I think it could be quite nice to give new users information about what site norms are and give a suggested spirit in which to engage with comments.

(Though I'm sure there's lots of things it'd be quite nice to tell new users about the spirit of the site, but there's of course bandwidth limitations on how much they'll read, so just because it's an improvement doesn't mean it's worth doing.)

6Said Achmiz2mo
If it’s worth banning[1] someone (and even urgently investing development resources into a feature that enables that banning-or-whatever!) because their comments might, possibly, on some occasions, potentially mislead users into falsely believing X… then it surely must be worthwhile to simply outright tell users ¬X? (I mean, of all the things that it might be nice to tell new users, this, which—if this topic, and all the moderators’ comments on it, are to be believed—is so consequential, has to be right up at the top of list?) -------------------------------------------------------------------------------- 1. Or rate-limiting, or applying any other such moderation action to. ↩︎

I have a version of heroic responsibility in my head that I don’t think causes one to have false beliefs about supernatural phenomena, so I’m interested in engaging on whether the version in my head makes sense, though I don’t mean to invalidate your strongly negative personal experiences with the idea.

I think there’s a difference between causing something and taking responsibility for it. There’s a notion of “I didn’t cause this mess but I am going to clean it up.” In my team often a problem arises that we didn’t cause and weren’t expecting. A few months ... (read more)

2TekhneMakre1mo
If I try to imagine what happened with jessicata, what I get is this: taking responsibility means that you're trying to apply your agency to everything; you're clamping the variable of "do I consider this event as being within the domain of things I try to optimize" to "yes". Even if you didn't even think about X before X has already happened, doesn't matter; you clamped the variable to yes. If you consider X as being within the domain of things you try to optimize, then it starts to make sense to ask whether you caused X. If you add in this "no excuses" thing, you're saying: even if supposedly there was no way you could have possibly stopped X, it's still your responsibility. This is just another instance of the variable being clamped; just because you supposedly couldn't do anything, doesn't make you not consider X as something that you're applying your agency to. (This can be extremely helpful, which is why heroic responsibility has good features; it makes you broaden your search, go meta, look harder, think outside the box, etc., without excuses like "oh but it's impossible, there's nothing I can do"; and it makes you look in retrospect at what, in retrospect, you could have done, so that you can pre-retrospect in the future.) If you're applying your agency to X "as though you could affect it", then you're basically thinking of X as being determined in part by your actions. Yes, other stuff makes X happen, but one of the necessary conditions for X to happen is that you don't personally prevent it. So every X is partly causally/agentially dependent on you, and so is partly your fault. You could have done more sooner.
1M. Y. Zuo2mo
  This sounds like a positive form of 'take responsibility' I can agree with. However, I'm not sure about this whole discussion in regards to 'the world', 'civilization', etc.  What does 'take responsibility' mean for an individual across the span of the entire Earth? For a very specific sub-sub-sub area, such as imparting some useful knowledge to a fraction of online fan-fiction readers of a specific fandom, it's certainly possible to make a tangible, measurable, difference, even without some special super-genius.  But beyond that I think it gets exponentially more difficult. Even a modestly larger goal of imparting some useful knowledge to a majority of online fan-fiction readers would practically be a life's effort, assuming the individual already has moderately above average talents in writing and so on.

My other comment notwithstanding, I do think the HPMOR quote is not very helpful for someone's mental health when they're in pain and seems a bit odd placed atop a section on advice, and I think the advice at the wrong time can feel oppressive. The hero-licensing post feels much less like it risks feeling oppressed by every bad thing that happens in the world. And personally I found Anna's post linked earlier to be much more helpful advice that is related to and partially upstream of the sorts of changes in my life that have reduced a lot of anxiety. If it... (read more)

I can understand thinking of yourself as having evil intentions, but I don't understand believing you're a partly-demonic entity. 

I think the way that the global market and culture can respond to ideas is strange and surprising, with people you don't know taking major undertakings based on your ideas, with lots of copying and imitation and whole organizations or people changing their lives around something you did without them ever knowing you. Like the way that Elon Musk met a girlfriend of his via a Roko's Basilisk meme, or one time someone on reddi... (read more)

6jessicata2mo
If you think you're responsible for everything, that means you're responsible for everything bad that happens. That's a lot of very bad stuff, some of which is motivated by bad intentions. An entity who's responsible for that much bad stuff couldn't be like a typical person, who is responsible for a modest amount of bad stuff. It's hard to conceptualize just how much bad stuff this hypothetical person is responsible for without supernatural metaphors; it's far beyond what a mere genocidal dictator like Hitler or Stalin is responsible for (at least, if you aren't attributing heroic responsibility to them). At that point, "well, I'm responsible for more bad stuff than I previously thought Hitler was responsible for" doesn't come close to grasping the sheer magnitude, and supernatural metaphors like God or Satan come closer. The conclusion is insane and supernatural because the premise, that you are personally responsible for everything that happens, is insane and supernatural. I'm not really sure how typical this particular response would be. But I think it's incredibly rare to actually take heroic responsibility literally and seriously. So even if I only rarely see evidence of people thinking they're demonic (which is surprisingly common, even if rare in absolute terms), that doesn't say much about the conditional likelihood of that response on taking heroic responsibility seriously.

The obvious dis-analogy is that if the police had no funding and largely ceased to exist, a string of horrendous things would quickly occur. Murders and thefts and kidnappings and rapes and more would occur throughout every country in which it was occurring, people would revert to tight-knit groups who had weapons to defend themselves, a lot of basic infrastructure would probably break down (e.g. would Amazon be able to pivot to get their drivers armed guards?) and much more chaos would ensue.

And if AI research paused, society would continue to basically function as it has been doing so far.

One of them seems to me like a goal that directly causes catastrophes and a breakdown of society and the other doesn't.

7Thomas Kwa2mo
Fair point. Another difference is that the pause is popular! 66-69% in favor of the pause, and 41% think AI would do more harm than good [https://www.monmouth.edu/polling-institute/reports/monmouthpoll_us_021523/] vs 9% for more good than harm.

Ray writes:

Here are some areas I think Said contributes in a way that seem important:

  • Various ops/dev work maintaining sites like readthesequences.com, greaterwrong.com, and gwern.com. 

For the record, I think the value here is "Said is the person independent of MIRI (including Vaniver) and Lightcone who contributes the most counterfactual bits to the sequences and LW still being alive in the world", and I don't think that comes across in this bullet.

4Raemon2mo
Yeah I agree with this, and agree it's worth emphasizing more. I'm updating the most recent announcement to indicate this more, since not everyone's going to read everything in this thread.

Okay! Good to know we concur on this. Was a bit worried, so thought I'd mention it.

Thanks!

I'm not sure if it's worth us having more back-and-forth, so I'll say my general feelings right now:

  • I think it's of course healthy and fine to have a bunch of major disagreements with Eliezer
  • I would avoid building "hate" toward him or building resentment as those things are generally not healthy for people to cultivate in themselves toward people who have not done evil things, as I think it will probably cause them to make worse choices by their own judgment
  • By-default do not count on anyone doing the hard work of making another forum for serious dis
... (read more)
4jacob_cannell2mo
Of course - my use of the word hate here is merely in reporting impressions from other ML/DL forums and the schism between the communities. I obviously generally agree with EY on many things, and to the extent I critique his positions here its simply a straightforward result of some people here assuming their correctness a priori.

I have not engaged much with your and Quintin's recent arguments about how deep learning may change the basic arguments, so I want to acknowledge that I would probably shift my opinion a bunch in some direction if I did. Nonetheless, a few related points:

  • I do want to say that on-priors the level of anger and antagonism that appears on most internet comment sections is substantially higher than what happens when the people meet in-person, and do not suspect a corresponding about of active antagonism would happen if Nate or Eliezer or John Wentworth went to
... (read more)
2jacob_cannell2mo
I just want to point out that seems like a ridiculous standard. Quintin's recent critique is not that dissimilar to the one I would write (and I already have spent some time trying to point out the various flaws in the EY/MIRI world model), and I expect that you would get many of the same objections if you elicited a number of thoughtful DL researchers. But few if any have been motivated - what's the point? Here's my critique in simplified form: the mainstream AI futurists (moravec,kurzweil,etc) predicted that AGI would be brain-like and thus close to a virtual brain emulation. Thus they were not so concerned about doom, because brain-like AGI seems like a more natural extension of humanity (moravec's book is named 'mind children' for a reason), and an easier transition to manage. In most ways that matter, Moravec/Kurzweil were correct, and EY was wrong. That really shouldn't be even up for debate at this point. The approach that worked - DL - is essentially reverse engineering the brain. This is in part due to how the successful techniques all ended up being directly inspired by neuroscience and the now proven universal learning & scaling hypotheses[1] (deep and or recurrent ANNs in general, sparse coding, normalization, relus, etc) OR indirectly recapitulated neural circuitry (transformer 'attention' equivalence to fast weight memory, etc). But in even simpler form: If you take a first already trained NN A and run it on a bunch of data and capture all its outputs, then train a second NN B on the input output dataset, the result is that B becomes a distilled copy - a distillation, of A. This is in fact how we train large scale AI systems. They are trained on human thoughts. -------------------------------------------------------------------------------- 1. The universal learning hypothesis is that the brain (and thus DL) uses simple universal learning algorithms, and all circuit content is learned automatically, which leads to the scaling hypothesis

I think Gerald is using 'posts' to mean any sort of content that has been 'posted', like when he writes "Obviously I have not done so or you would not be reading this post" referring to the comment he has written.

2habryka2mo
Ah, that makes sense. 

If you search by top-comments (as you can on GreaterWrong) you can find this 55 karma comment by Gerald Monroe from 2 years ago.

(The next highest karma comment is 22, 20, 19, so there's only the one comment I would call 'highly upvoted'.)

2habryka2mo
Sure, though I was specifically asking about posts, which Gerald mentioned. 

Can I ask what your epistemic state here is exactly? Here are some options:

  • The arguments Eliezer put forward do not clearly apply to Deep Learning and therefore we don't have any positive reason to believe that alignment will be an issue in ML
  • The arguments Eliezer put forward never made sense in the first place and therefore we do not have to worry about the alignment problem
  • The arguments Eliezer put forward captured a bunch of important things about the alignment problem but due to some differences in how we get to build ML systems we actually know of a p
... (read more)
2jacob_cannell2mo
Yes but does not follow. Yes (for some of the arguments), but again: does not follow. Yes - such as the various more neuroscience/DL inspired approaches (Byrnes, simboxes, shard theory, etc.), or others a bit harder to categorize like davdidad's approach, or external empowerment. But also I should point out that RLHF may work better for longer than most here anticipate, simply because if you distill the (curated) thoughts of mostly aligned humans you may just get mostly aligned agents.

Yep! By-default your own comments start at either +1 or +2, and your own posts start with +n. Adds a bit of default sorting into the site. Pretty small effect IMO. (Announcement of strong upvoting is here.)

Also, can I just remind you that for most of LessWrong's history the top-karma post was Holden's critique of SingInst where he recommended against funding SingInst and argued in favor of Tool AI as the solution. Recently Eliezer's List-of-Lethalities became the top-karma post, but less than a month later Paul's response-and-critique post became the top-karma post where he argued that the problem is much more tractable than Eliezer thinks, and generally advocates a very different research strategy for dealing with alignment. 

Eliezer is the primary pers... (read more)

So really my disagreement is more on alignment strategy. A problem with this site is that it overweights EY/MIRI classic old alignment literature and arguments by about 100x what it should be

I don't think there are many people with alignment strategies and research that they're working on. Eliezer has a hugely important perspective, Scott Garrabrant, Paul Christiano, John Wentworth, Steve Byrnes, and more, all have approaches and perspectives too that they're working full-time on. I think if you're working on this full-time and any of your particular ideas... (read more)

So really my disagreement is more on alignment strategy. A problem with this site is that it overweights EY/MIRI classic old alignment literature and arguments by about 100x what it should be

I don't think there are many people with alignment strategies and research that they're working on.

I agree that's a problem - but causally downstream of the problem I mention. Whereas Bostrom deserves credit for raising awareness of AI-risk in academia, EY/MIRI deserves credit for awakening many young techies to the issue - but also some blame.

Whether intentio... (read more)

4Ben Pace2mo
Also, can I just remind you that for most of LessWrong's history the top-karma post was Holden's critique of SingInst where he recommended against funding SingInst and argued in favor of Tool AI as the solution. Recently Eliezer's List-of-Lethalities became the top-karma post, but less than a month later Paul's response-and-critique post became the top-karma post where he argued that the problem is much more tractable than Eliezer thinks, and generally advocates a very different research strategy for dealing with alignment.  Eliezer is the primary person responsible for noticing and causing people to work on the alignment problem, due to his superior foresight and writing skill, and also founded this site, so most people here have read his perspective and understand it somewhat, but any notion that dissent isn't welcomed here (which I am perhaps over-reading into your comment) seems kind of obviously not the case.

Friends aren’t resources for intellectual stimulation or new insights. I don’t want my friends to like me because I read niche blogs or have things to say about crypto. It comes dangerously close to conflating knowing a lot, reading a lot, or having thoughtful things to say with moral goodness.

I think I would reverse most sentences in this paragraph. Being able to think for yourself and have your own useful and insightful takes on how the world is working seems to me closer to a requirement for being able to take morally good action. Anything can be g... (read more)

Theatre, lectures, debates, live music... I'm not saying that all of these feel the same when recorded, but I am saying that many of them can be improved when done naturalistically and audience-less, and more importantly they have a much lesser effect of wasting the time of the audience, which in my opinion happens for the majority of the audience to a substantial degree.

I don't think it's mildly insulting, I think it's ambiguously insulting, in that a person wanting to insult you might do it. But in general I think it's a totally reasonable question in truth-seeking and I'd be sad if people required disclaimers to clarify that it isn't meant insultingly, just to ask for examples of what the person is talking about.

(Commenting from Recent Discussion)

(Commenting from recent discussion, also intended as a reply to Gwern)

The annual review is an attempt to figure out what were the best contributions with the benefit of a great deal of hindsight, and I think it's prosocial to contribute to it, similar to how it was prosocial to contribute to the LW survey back when Scott ran a big one every year.

I am always pleased when people contribute, and sometimes I am sad if there are particular users whose reviews I'd really like to read but don't write any. But I don't think anyone is obligated to write reviews!

Really great post.

The two parts that stand out most to me are the Causality and Ontology Change sections.

Regarding Causality, I agree that there will be little investment into robotics as a mechanism for intervening on the world and building causal models. However, I don't see why practicing on videogames doesn't produce this sort of interventionist data and why AIs wouldn't learn causal models from that. And it doesn't seem that expensive to create the data. It's already happened a bunch with AIs trained on mutltiplayer videogames, and later on it will ge... (read more)

It's... possible this is actually the single best example of a public doublecrux writeup that I know of?

This sentence was confusing to me given that the post does not mention 'double crux', but I mentioned it to someone and they said to think of it as the mental motion and not the explicit format, and that makes more sense to me.

2Raemon2mo
Yeah that's what I intended.

I strong upvoted this quite interesting post, but I want to mention that I do not mean my upvote to endorse this particular historical story of Socrates. This is because I have not read any Socrates nor read about his history, and I do not personally have confident beliefs about whether the Socrates in this post is the same as the historic one. (The list of historic details about Athens did me help understand the environment though.)

Edit: Upvote retracted, I realize this is a post criticizing Said that he cannot comment on.

3[DEACTIVATED] Duncan Sabien2mo
It is not. It is, in fact, a generalized and timeless stance that I believe in, independent of Said, regardless of the fact that recent interactions with Said were among the more recent pushes for me to move it from somewhere-in-the-middle-of-my-pile-of-70-essays-to-write up to the top. (A commenter on FB pointed out that I had formed the seed of this essay two years ago [https://www.facebook.com/duncan.sabien/posts/pfbid02wgp7FYZURwrdCZBLsj5FaVUVsc32WeY1RJe8Qf6pitc1zbwKb2gwZ8YMAdnYGiTHl], probably around the time of Vaniver's original comment, which was not about Said.) Other commenters immediately recognizing Said in the content is not, I think, something that should result in your removing an upvote. And I also don't think you would generally endorse a position of something like "don't try to extract timeless or generalized lessons from specific experiences." Said being an instance of a dynamic doesn't mean describing the dynamic (without specific reference to him!) is bad. I don't know who's enacting the psy-op; maybe it's just society-at-large, but I claim "force this to be interpreted as an attack on a specific individual such that it now feels morally wrong to upvote it" is a psy-op.

Ah good, I thought you were proposing a drastically higher bar.

shift the default assumptions of LessWrong to "users by default have a rate limit of 1-comment-per day"

Natural times I expect this to be frustrating are when someone's written a post, got 20 comments, and tries to reply to 5 of them, but is locked after the first one. 1 per day seems too strong there. I might say "unlimited daily comments on your own posts".

I also think I'd prefer a cut-off where after which you're trusted to comment freely. Reading the positive-selection post (which I agree with), I think some bars here could include having written a curated post or a post with 200+ karma or having 1000 karma on your account.

4Raemon2mo
I'm not particularly attached to these numbers, but fyi the scale I was originally imagining was "after the very first upvote, you get something like 3 comments a day, and after like 5-10 karma you don't have a rate limit." (And note, initially you get one post and one comment, so you get to reply to your post's first comment) I think in practice, in the world where you receive 4 comments but a) your post hasn't been upvoted much and b) none of your responses to the first three comments didn't get upvoted, my expectation is you're a user I'd indeed prefer to slow down, read up on site guidelines and put more effort into subsequent comments. I think having 1000 karma isn't actually a very high bar, but yeah I think users with 2+ posts that either have 100+ karma or are curated, should get a lot more leeway.
Load More