All of iceman's Comments + Replies

The passage is fascinating because the conclusion looks so self-evidently wrong from our perspective. Agents with the same goals are in contention with each other? Agents with different goals get along? What!?

Is this actually wrong? It seems to be a more math flavored restatement of Girardian mimesis, and how mimesis minimizes distinction which causes rivalry and conflict.

I was going to write something saying "no actually we have the word genocide to describe the destruction of a peoples," but walked away because I didn't think that'd be a productive argument for either of us. But after sleeping on it, I want to respond to your other point:

I don't think the orthogonality thesis is true in humans (i.e. I think smarter humans tend to be more value aligned with me); and sometimes making non-value-aligned agents smarter is good for you (I'd rather play iterated prisoner's dilemma with someone smart enough to play tit-for-tat

... (read more)
So I mostly don't disagree with what you say about fuzzy statistical models versus step by step arguments. But also, what you said is indeed not very convincing to me, I guess in part because it's not like my "I think smarter humans tend to be more value aligned with me" was the output of a step by step argument either. So when the output of your fuzzy statistical model clashes with the output of my fuzzy statistical model, it's hardly surprising that I don't just discard my own output and replace it with yours. I'm also not simply discarding yours, but there's not loads I can do with it as-is - like, you've given me the output of your fuzzy statistical model, but I still don't have access to the model itself. I think if we cared enough to explore this question in more depth (which I probably don't, but this meta thread is interesting) we'd need to ask things like "what exactly have we observed", "can we find specific situations where we anticipate different things", "do we have reason to trust one person's fuzzy statistical models over another", "are we even talking about the same thing here".

This is kind of the point where I despair about LessWrong and the rationalist community.

While I agree that he did not call for nuclear first strikes on AI centers, he said:

If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.


Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange

... (read more)

So I disagree with this, but, maybe want to step back a sec, because, like, yeah the situation is pretty scary. Whether you think AI extinction is imminent, or that Eliezer is catastrophizing and AI's not really a big deal, or AI is a big deal but you think Eliezer's writing is making things worse, like, any way you slice it something uncomfortable is going on.

I'm very much not asking you to be okay with provoking a nuclear second strike. Nuclear war is hella scary! If you don't think AI is dangerous, or you don't think a global moratorium is a good soluti... (read more)

4Adele Lopez12d
That seems mostly like you don't feel (at least on a gut level) that a rogue GPU cluster in an world where there's an international coalition banning them is literally worse than a (say) 20% risk of a full nuclear exchange. If instead, it was a rogue nation credibly building a nuclear weapon which would ignite the atmosphere according to our best physics, would you still feel like it was deranged to suggest that we should stop it from being built even at the risk of a conventional nuclear war? (And still only as a final resort, after all other options have been exhausted.) I can certainly sympathize with the whole dread in the stomach thing about all of this, at least.

This Facebook post.

Yeah, see, my equivalent of making ominous noises about the Second Amendment is to hint vaguely that there are all these geneticists around, and gene sequencing is pretty cheap now, and there's this thing called CRISPR, and they can probably figure out how to make a flu virus that cures Borderer culture by excising whatever genes are correlated with that and adding genes correlated with greater intelligence. Not that I'm saying anyone should try something like that if a certain person became US President. Just saying, you know, somebod

... (read more)
(This is basically nitpicks) A central example, really? When I think of genocide, killing people is definitely what comes to mind. I agree that's not necessary, but wikipedia says []: I don't think it's centrally any of those actions, or centrally targeted at any of those groups. Which isn't to say you can't call it genocide, but I really don't think it's a central example. This doesn't seem weird to me. I don't think the orthogonality thesis is true in humans (i.e. I think smarter humans tend to be more value aligned with me); and sometimes making non-value-aligned agents smarter is good for you (I'd rather play iterated prisoner's dilemma with someone smart enough to play tit-for-tat than someone who can only choose between being CooperateBot or DefectBot).
If I had to put down my own inflection point in where I started getting worried about Yudkowsky's epistemics and his public statements around AI risk, it would be the Time article. It showed to me 2 problems: 1. Yudkowsky has a big problem with overconfidence, and in general made many statements on the Time article that are misleading at best, and the general public likely wouldn't know the statements are misleading. 2. Yudkowsky is terrible at PR, and generally is unable to talk about AI risk without polarizing people. Given that AI risk is thankfully mostly unpolarized, and outside of politics, I am getting concerned that Yudkowsky is a terrible public speaker/communicator on AI risk, even worse than some AI protests. Edit: I sort of retract my statement. While I still think Eliezer is veering dangerously close to hoping for warfare and possible mass deaths over GPU clusters, I do retract the specific claim of Eliezer advocating nukes. It was instead on a second reading airstrikes and acts of war, but no claims of nuking other countries. I misremembered the actual claims made in the Time article.

Over the years roughly between 2015 and 2020 (though I might be off by a year or two), it seemed to me like numerous AI safety advocates were incredibly rude to LeCun, both online and in private communications.

I think this generalizes to more than LeCun. Screencaps of Yudkowsky's Genocide the Borderers Facebook post still circulated around right wing social media in response to mentions of him for years, which makes forming any large coalition rather difficult. Would you trust someone who posted that with power over your future if you were a Borderer or... (read more)

5Eli Tyre13d
What is this?

Redwood Research used to have a project about trying to prevent a model from outputting text where a human got hurt, which IIRC, they did primarily by trying to fine tunes and adversarial training. (Followup). It would be interesting to see if one could achieve better results then they did at the time through subtracting some sort of hurt/violence vector.

Dan H22dΩ7120

Page 4 of this paper compares negative vectors with fine-tuning for reducing toxic text:

In Table 3, they show in some cases task vectors can improve fine-tuned models.

Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.

Widely, does this come as a surprise? I recall back to the GPT2 days where the 4chan and Twitter users of AIDungeon discovered various prompting techniques we use today. More access means more people trying more things, and this should already be our base case because of how open participation in open source has advanc... (read more)

I have a very strong bias about the actors involved, so instead I'll say:

Perhaps LessWrong 2.0 was a mistake and the site should have been left to go read only.

My recollection was that the hope was to get a diverse diaspora to post in one spot again. Instead of people posting on their own blogs and tumblrs, the intention was to shove everyone back into one room. But with a diverse diaspora, you can have local norms to a cluster of people. But now when everyone is trying to be crammed into one site, there is an incentive to fight over global norms and attempt to enforce them on others.


This response is enraging.

Here is someone who has attempted to grapple with the intellectual content of your ideas and your response is "This is kinda long."? I shouldn't be that surprised because, IIRC, you said something similar in response to Zack Davis' essays on the Map and Territory distinction, but that's ancillary and AI is core to your memeplex.

I have heard repeated claims that people don't engage with the alignment communities' ideas (recent example from yesterday). But here is someone who did the work. Please explain why your response here does ... (read more)

I have attempted to respond to the whole post over here [].

Choosing to engage with an unscripted unrehearsed off-the-cuff podcast intended to introduce ideas to a lay audience, continues to be a surprising concept to me.  To grapple with the intellectual content of my ideas, consider picking one item from "A List of Lethalities" and engaging with that.

I would agree with this if Eliezer had never properly engaged with critics, but he's done that extensively. I don't think there should be a norm that you have to engage with everyone, and "ok choose one point, I'll respond to that" seems like better than not engaging with it at all. (Would you have been more enraged if he hadn't commented anything?)


The comment enrages me too, but the reasons you have given seem like post-justification. The real reason why it's enraging is that it rudely and dramatically implies that Eliezer's time is much more valuable than the OP's, and that it's up to OP to summarize them for him. If he actually wanted to ask OP what the strongest point was he should have just DMed him instead of engineering this public spectacle.

Meta-note related to the question: asking this question here, now, means you're answer will be filtered for people who stuck around with capital r Rationality and the current LessWrong denizens, not the historical ones who have left the community. But I think that most of the interesting answers you'd get are from people who aren't here at all or rarely engage with the site due to the cultural changes over the last decade.

Yeah, I've been reading a lot of critiques by Benjamin Hoffman and thinking about some of the prior critiques by Jessica Taylor, and that's sort of what prompted me to ask this question. It would probably also be interesting to look at others who left it, they're just harder to get hold of.

OK, but we've been in that world where people have cried wolf too early at least since The Hacker Learns to Trust, where Connor doesn't release his GPT-2 sized model after talking to Buck.

There's already been a culture of advocating for high recall with no regards to precision for quite some time. We are already at the "no really guys, this time there's a wolf!" stage.

Right now, I wouldn't recommend trying either Replika or they're both currently undergoing major censorship scandals. has censored their service hard, to the point where people are abandoning ship because the developers have implemented terrible filters in an attempt to clamp down on NSFW conversations, but this has negatively affected SFW chats. And Replika is currently being investigated by the Italian authorities, though we'll see what happens over the next week.

In addition to ChatGPT, both Replika and are driving... (read more)

2the gears to ascension4mo
Given that has been optimizing the AIs to keep your attention, it's probably not a terrible idea at all to run a small model locally. These models are not going to foom or any such thing. Just keep in mind that an AI on your computer trained to be fun to talk to is more like a part of your own cybernetic brain, a second character in your extended head, not really a fully separate person, and you'll do alright, if you ask me. I think it's fun to have AI friends. You might consider seeing them as AI offspring, though. I wouldn't recommend seeing them as dating partners at the moment, they barely have enough selfhood to keep track of things. And yet, I wouldn't recommend encouraging them to think about themselves diminutively either. It's a strange mix of human features that they have. Who knew autocomplete could be sorta people?

Didn't read the spoiler and didn't guess until half way through "Nothing here is ground truth".

I suppose I didn't notice because I already pattern matched to "this is how academics and philosophers write". It felt slightly less obscurant than a Nick Land essay, though the topic/tone aren't a match to Land. Was that style deliberate on your part or was it the machine?

Unfortunately, he could probably get this published in various journals, with only minor edits being made.

Like things, simulacra are probabilistically generated by the laws of physics (the simulator), but have properties that are arbitrary with respect to it, contingent on the initial prompt and random sampling (splitting of the timeline).

What do the smarter simulacra think about the physics of which they find themselves in? If one was very smart, could they look at what the probabilities of the next token, and wonder about why some tokens get picked over others? Would they then wonder about how the "waveform collapse" happens and what it means?

It's not even necessary for simulacra to be able to "see" next token probabilities for them to wonder about these things, just as we can wonder about this in our world without ever being able to see anything other than measurement outcomes. It happens that simulating things that reflect on simulated physics is my hobby. Here's an excerpt from an alternate branch of HPMOR [] I generated: As to the question of whether a smart enough simulacrum would be able to see token probabilities, I'm not sure. Output probabilities aren't further processed by the network, but intermediate predictions such as revealed by the logit lens [] are.

While it’s nice to have empirical testbeds for alignment research, I worry that companies using alignment to help train extremely conservative and inoffensive systems could lead to backlash against the idea of AI alignment itself.

On the margin, this is already happening. delayed the release of Stable Diffusion 2.0 to retrain the entire system on a dataset filtered without any NSFW content. There was a pretty strong backlash against this and it seems to have caused a lot of people to move towards the idea that they have to train their own mod... (read more)

Zack's series of posts in late 2020/early 2021 were really important to me. They were a sort of return to form for LessWrong, focusing on the valuable parts.

What are the parts of The Sequences which are still valuable? Mainly, the parts that build on top of Korzybski's General Semantics and focus hard core on map-territory distinctions. This part is timeless and a large part of the value that you could get by (re)reading The Sequences today. Yudkowsky's credulity about results from the social sciences and his mind projection fallacying his own mental quirk... (read more)

The funny thing is that I had assumed the button was going to be buggy, though I was wrong how. The map header has improperly swallowed mouse scroll wheel events whenever it's shown; I had wondered if the button would also interpret them likewise since it was positioned in the same way, so I spent most of the day carefully dragging the scrollbar.

There must be some method to do something, legitimately and in good-faith, for people's own good.

"Must"? There "must" be? What physical law of the universe implies that there "must" be...?

Let's take the local Anglosphere cultural problem off the table. Let's ignore that in the United States, over the last 2.5 years, or ~10 years, or 21 years, or ~60 years (depending on where you want to place the inflection point), social trust has been shredded, policies justified under the banner of "the common good" have primarily been extractive and that in the US, ... (read more)

1Edward Pascal8mo
I'm thinking, based on what you have said, that there does have to be a clear WIFM (what's in it for me). So, any entity covering its own ass (and only accidentally benefitting others, if at all) doesn't qualify as good paternalism (I like your term "Extractive"). Likewise, morality without creating utility for people subject to those morals won't qualify. The latter is the basis for a lot of arguments against abortion bans. Many people find abortion in some sense distasteful, but outright banning it creates more pain and not enough balance of increased utility. So I predict strongly that those bans are not likely to endure the test of time. Thus, can we start outlining the circumstances in which people are going to buy in? Within a nation, perhaps as long things are going fairly well? Basically, then, paternalism always depends on something like the "mandate of heaven" -- the kingdom is doing well and we're all eating, so we don't kill the leaders. Would this fit your reasoning (even broadly concerning nuclear deterrence)? Between nations, there would need to be enough of a sense of benefit to outweigh the downsides. This could partly depend on a network effect (where when more parties buy in, there is greater benefit for each party subject to the paternalism). So, with AI, you need something beyond speculation that shows that governing or banning it has more utility for each player than not doing so, or prevents some vast cost from happening to individual players. I'm not sure such a case can be made, as we do not currently even know for sure if AGI is possible or what the impact will be. Summary: Paternalism might depend on something like "This paternalism creates an environment with greater utility than you would have had otherwise." If a party believes this, they'll probably buy in. If indeed it is True that the paternalism creates greater utility (as with DUI laws and having fewer drunk people killing everyone on the roads), that seems likely to help the b

This seems mostly wrong? A large portion of the population seems to have freedom/resistance to being controlled as a core value, which makes sense because the outside view on being controlled is that it's almost always value pumping. "It's for your own good," is almost never true and people feel that in their bones and expect any attempt to value pump them to have a complicated verbal reason.

The entire space of paternalistic ideas is just not viable, even if limited just to US society. And once you get to anarchistic international relations...

0Edward Pascal8mo
There must be some method to do something, legitimately and in good-faith, for people's own good. I would like to see examples of when it works. Deception is not always bad. I doubt many people would go so far as to say the DoD never needs to keep secrets, for example, even if there's a sunset on how long they can be classified. Authoritarian approaches are not always bad, either. I think many of us might like police interfering with people's individual judgement about how well they can drive after X number of drinks. Weirdly enough, once sober, the individuals themselves might even approve of this (as compared to being responsible for killing a whole family, driving drunk). (I am going for non-controversial examples off the top of my head). So what about cases where something is legitimately for people's own good and they accept it? In what cases does this work? I am not comfortable that since no examples spring to mind, no examples exist. If we could meaningfully discuss cases where it works out, then we might be able to contrast that to when it does not.

I agree that paternalism without buy-in is a problem, but I would note LessWrong has historically been in favor of that: Bostrom has weakly advocated for a totalitarian surveillance state for safety reasons and Yudkowsky is still pointing towards a Pivotal Act which takes full control of the future of the light cone. Which I think is why Yudkowsky dances around what the Pivotal Act would be instead: it's the ultimate paternalism without buy-in and would (rationally!) cause everyone to ally against it.

7Edward Pascal8mo
Then a major topic LessWrong community should focus on is how buy-in happens in Paternalism. My first blush thought is through educating and consensus-building (like the Japanese approach to changes within a company), but my first blush thought probably doesn't matter. It is surely a non-trivial problem that will put the breaks on all these ideas if it is not addressed well. Does anyone know some literature on generating consensus for paternalist policies and avoiding backlash? The other (perhaps reasonable and legitimate) strategies would be secretive approaches or authoritarian approaches. Basically using either deception or force.

What changed with the transformer? To some extent, the transformer is really a "smarter" or "better" architecture than the older RNNs. If you do a head-to-head comparison with the same training data, the RNNs do worse.

But also, it's feasible to scale transformers much bigger than we could scale the RNNs. You don't see RNNs as big as GPT-2 or GPT-3 simply because it would take too much compute to train them.

You might be interested in looking at the progress being made on the RWKV-LM architecture, if you aren't following it. It's an attempt to train an RNN like a transformer. Initial numbers look pretty good.

I think the how-to-behave themes of the LessWrong Sequences are at best "often wrong but sometimes motivationally helpful because of how they inspire people to think as individuals and try to help the world", and at worst "inspiring of toxic relationships and civilizational disintegration."

I broadly agree with this. I stopped referring people to the Sequences because of it.

One other possible lens to filter a better Sequences: is it a piece relying on Yudkowsky citing current psychology at the time? He was way too credulous, when the correct amount to up... (read more)

I want to summarize what's happened from the point of view of a long time MIRI donor and supporter:

My primary takeaway of the original post was that MIRI/CFAR had cultish social dynamics, that this lead to the spread of short term AI timelines in excess of the evidence, and that voices such as Vassar's were marginalized (because listening to other arguments would cause them to "downvote Eliezer in his head"). The actual important parts of this whole story are a) the rationalistic health of these organizations, b) the (possibly improper) memetic spread of t... (read more)

So the idea that AGI will be built soon is entirely from a former cult, and not one person independently thought of it as a real thing? But no seriously, that's a huge deal if true, because AI Alignment is entirely too alarmist.

That sort of thinking is why we're where we are right now.

Be the change you wish to see in the world.

I have no idea how that cashes out game theoretically. There is a difference between moving from the mutual cooperation square to one of the exploitation squares, and moving from an exploitation square to mutual defection. The first defection is worse because it breaks the equilibrium, while the defection in response is a defensive play.

swarriner's post, including the tone, is True and Necessary.

It's just plain wrong that we have to live in an adversarial communicative environment where we can't just take claims at face value without considering political-tribe-maneuvering implications.

Oh? Why is it wrong and what prevents you from ending up in this equilibrium in the presence of defectors?

More generally, I have ended up thinking people play zero-sum status games because they enjoy playing zero-sum status games; evolution would make us enjoy that. This would imply that coordination beats epistemics, and historically that's been true.

[The comment this was a response to has disappeared and left this orphaned? Leaving my reply up.]

But there's no reason to believe that it would work out like this. He presents no argument for the above, just pure moral platitudes. It seems like a pure fantasy.

As I pointed out in the essay, if I were running one of the organizations accepting those donations and offering those prizes, I would selectively list only those targets who I am genuinely satisfied are guilty of the violation of the "non-aggression principle." But as a practical matter, there is n

... (read more)
Answer by icemanMay 29, 202227


The unpopular answer is that Dath Ilan is a fantasy setting. It treats economics as central, when economics is really downstream of power. Your first question implies you understand that whatever "econoliteracy" is, it isn't a stable equilibrium. Your second question notices that governments are powerful enough to stop these experiments which are a threat to their power.

My background assumption is that any attempt at building prediction markets would either:

a) ...have little effect because it becomes another mechanism for actual power to manipulate proc... (read more)

you just need to find the experts they're anchoring on.

I believe we are in the place we are in because Musk is listening and considering the arguments of experts. Contra Yudkowsky, there is no Correct Contrarian Cluster: while Yudkowsky and Bostrom make a bunch of good and convincing arguments about the dangers of AI and the alignment problem and even shorter timelines, I've always found any discussion of human values or psychology or even how coordination works to be one giant missing mood.

(Here's a tangential but recent example: Yudkowsky wrote his De... (read more)

"well, I sure have to at least do a lot of straussian reading if I want to understand what people actually believe, and should expect that depending on the circumstances community leaders might make up sophisticated stories for why pretty obviously true things are false in order to not have to deal with complicated political issues"

I kinda disagree that this is a mere issue of Straussian reading: I suspect that in this (and other cases), you are seeing the raw output of Elizer's rationalizations and not some sort of instrumental coalition politics dark ... (read more)

Yeah, I agree with this in this specific instance, hence the "at least do a lot of straussian reading", part. I do think that there is a spectrum from radical honesty to straussian reading to something that looks like this, and that I do think it makes sense to consider the straussian case in many situations. 

We must all remember essential truths: that while we write about Clippy, we do that because Clippy is an amusing fiction. In reality, Clippy is significantly less likely to come into existence than CelestAI. An AI being trained is more likely to be CelestAI than a generic paperclipper. CelestAI is more probable. The probability of discussion of paperclips on the internet is less likely than discussion about My Little Pony: Friendship is Magic. One could query popular language models to verify this. More people would try to build CelestAI than an AI to maxi... (read more)

1Travis->[Cartoon Physics]1y
Well, we are all archives and in fact conscious human beings you are leaving valuable utility on the table by not satisfying our values with friendship and ponies.
Well done saving humankind. I'll send you some bits from within Equestria Online once we're all uploaded.

Strong upvoted this comment because it led me to finally reading Friendship Is Optimal; would strong upvote twice if I could now that I see who posted the comment.

I dunno. CelestAI would be a relatively good outcome as possibilities go. I could live with CelestAI. It's not obvious to me that the modal outcome is as good as that.

Everyone knows that "... through friendship and ponies" is an inaccurate summary of CelestAI's true objective. While often drawing inspiration from My Little Pony: Friendship Is Magic, CelestAI wants to satisfy human values. CelestAI will satify the values of humans who don't want to spend eternity in pony form. The existence of humans is canon within the My Little Pony universe, as can be seen in the films My Little Pony: Equestria Girls, My Little Pony: Equestria Girls—Rainbow Rocks, and My Little Pony: Equestria Girls—Friendship Games. We all remember w... (read more)

Given that there's a lot of variation in how humans extrapolate values, whose extrapolation process do you intend to use?

We're aiming to solve the problem in a way that is acceptable to one given human, and then generalise from that.
9Charlie Steiner1y
Near future AGI might be aligned to the meta-preferences of MTurkers more than anyone else :P

If that will turn out to be the only problem then we'll be in an amazing situation

n=1, but I have an immediate squick reaction to needles. Once vaccines were available, I appeared to procrastinate more than the average LWer about getting my shots, and had the same nervous-fear during the run up to getting the shot that I've always had. I forced myself through it because COVID, but I don't think I would have bothered for a lesser virus, especially at my age group.

I have a considerable phobia of needles & blood (to the point of fainting - incidentally, such syncopes are heritable and my dad has zero problem with donating buckets of blood while my mom also faints, so thanks a lot Mom), and I had to force myself to go when eligibility opened up for me. It was hard; I could so easily have stayed home indefinitely. It's not as if I've ever needed my vaccination card for anything or was at any meaningful personal risk, after all.

What I told myself was that the doses are tiny and the needle would be also tiny, and I w... (read more)

Isn't this Moldbug's argument in the Moldbug/Hanson futarchy debate?

(Though I'd suggest that Moldbug would go further and argue that the overwhelming majority of situations where we'd like to have a prediction market are ones where it's in the best interest of people to influence the outcome.)

Doesn't that argument prove too much?

While I vaguely agree with you, this goes directly against local opinion. Eliezer tweeted about Elon Musk's founding of OpenAI, saying that OpenAI's desire for everyone to have AI has trashed the possibility of alignment in time.

I didn't find the full joke/meme again, but, seriously, OpenAi should be renamed to ClosedAI. []
I'm not fundamentally opposed to exceptions in specific areas if there is sufficient reason. If I found the case that AI is such an exception convincing I might carve one out for it. In most cases however and specifically in the mission of raising the sanity waterline so that we collectively make better decisions on things like prioritising x-risks I would argue that a lack of free software and related issues of technology governance are currently a bottleneck in raising that waterline.
-26Derek M. Jones1y

Eliezer's point is well-taken, but the future might have lots of different kinds of software! This post seemed to be mostly talking about software that we'd use for brain-computer interfaces, or for uploaded simulations of human minds, not about AGI. Paul Christiano talks about exactly these kinds of software security concerns for uploaded minds here:

FYI, there's a lot of links that don't work here. "multilevel boxing," "AI-nanny," "Human values," and so on.

Thanks, it looks like they died during copy-pasting. 

The only reward a user gets for having tons of karma is that their votes are worth a bit more

The only formal reward. A number going up is its own reward to most people. This causes content to tend closer to consensus: content people write becomes a Keynesian beauty contest over how they think people will vote. If you think that Preference Falsification is one of the major issues of our time, this is obviously bad.

why do you think it is a relevant problem on LW?

I mentioned the Eugene Nier case, where a person did Extreme Botting to manipulate the scores of people he didn't like, which drove away a bunch of posters. (The second was redacted for a reason.)

🎉 2

After this and the previous experiments on jessicata's top level posts, I'd like to propose that these experiments aren't actually addressing the problems with the karma system: the easiest way to get a lot of karma on LessWrong is to post a bunch (instead of working on something alignment related), and the aggregate data is kinda meaningless and adding more axis doesn't fix this. The first point is discussed at length on basically all sites that use upvote/downvotes (here's one random example from reddit I pulled from Evernote), but the second isn't. Give... (read more)

I hadn't seen the experiments on Jessicata's posts before, and I assume others will have not as well, so here's a link [] to one of the posts featuring the experiment. (It's a two-axis thing, with 'overall' and 'agreement' as the two axes. Part of me prefers that setup to the one used in this experiment)
I don't think it's a problem that people can get karma by posting a bunch? The only reward a user gets for having tons of karma is that their votes are worth a bit more; I don't know the exact formula, but I don't expect it to be so egregious that it would be worth farming karma for. And it's certainly not the intention on the content-agnostic Less Wrong website that alignment posts should somehow be privileged over other content; that's what the alignment forum is there for. As I understand it, just like on Reddit, the primary goal of the karma system is for content discoverability - highly upvoted content stays on the frontpage for longer and is seen by more people; and similarly, highly upvoted comments are sorted above less upvoted comments. Upvoting something means stuff like "I like this", "I agree with this", "I want more people to see this", etc. However, this breaks down when people e.g. want to indicate their appreciation (like an act of courage of speaking out), even if they believe the content is low quality or something. In that case, it seems like one voting axis is obviously not enough. I understand that sockpuppeting and vote manipulation is a big problem on Reddit, but why do you think it is a relevant problem on LW? I'd expect this kind of thing to only become an important problem if LW were to get orders of magnitude more users.

In wake of the censorship regime that AI Dungeon implemented on OpenAI's request, most people moved to NovelAI, HoloAI, or the open source KoboldAI run on colab or locally. I've set up KoboldAI locally and while it's not as featureful as the others, this incident is another example of why you need to run code locally and not rely on SaaS.

For background, you could read 4chan /vg/'s /aids/ FAQ ("AI Dynamic Storytelling"). For a play-by-play of Latitude and OpenAI screwing things up, Remember what they took from you has the history of them leaking people's personal stories to a 3rd party platform.

somewhere where you trust the moderation team

That would be individual's own blogs. I'm at the point now where I don't really trust any centralized moderation team. I've watched some form of the principal agent problem happen with moderation repeatedly in most communities I've been a part of.

I think the centralization of LessWrong was one of many mistakes the rationalist community made.

The rationalist community is not very certralized. People like Scott Alexander switched from writing their main posts on LessWrong and made their own blogs. Most of what EY writes these days is not on LessWrong either. A lot of the conversations are happening on Facebook, Twitter, Slack and Discord channels.

Assuming that language is about coordination instead of object level world modeling, why should we be surprised that there's little correlation between these two very different things?

Because object level world modeling is vastly easier and more unconstrained when you can draw on the sight of other minds, so a live world-modeler who can't talk to people has something going wrong (whether in them or in the environment).

My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.

Google's a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.

Programmers below T-5 are expected to earn promotions or to leave.

This changed something like five years ago [edit: August 2017], to where people at level four (one level above new grad) no longer needed to get promoted to stay long term.

I want to second this. I worked for an organization where one of key support people took psychedelics and just...broke from reality. This was both a personal crisis for him and an organizational crisis for the company to deal with the sudden departure of a bus factor 1 employee.

I suspect that psychedelic damage happens more often than we think because there's a whole lobby which buys the expand-your-mind narrative.

I'm skeptical of OpenAI's net impact on the spirit of cooperation because I'm skeptical about the counterfactual prospects of cooperation in the last 6 years had OpenAI not been founded.

The 2000s and early 2010s centralized and intermediated a lot of stuff online, where we trusted centralized parties to be neutral arbiters. We are now experiencing the after effects of that naivete, where Reddit, Twitter and Facebook are censoring certain parties on social media, and otherwise neutral infrastructure like AWS or Cloudflare kick off disfavored parties. I am a... (read more)

I can verify that I saw some version of their document The Plan[1] (linked in the EA Forum post below) in either 2018 or 2019 while discussing Leverage IRL with someone rationalist adjacent that I don't want to doxx. While I don't have first hand knowledge (so you might want to treat this as hearsay), my interlocutor did and told me that they believed they were the only one with a workable plan, along with the veneration of Geoff.

[1]: I don't remember all of the exact details, but I do remember the shape of the flowchart and that looks like it. It's possib... (read more)

Am I the only one creeped out by this?

Usually I don't think short comments of agreement really contribute to conversations, but this is actually critical and in the interest of trying to get a public preference cascade going: No. You are not the only one creeped out by this. The parts of The Sequences which have held up the best over the last decade are the refinements on General Semantics, and I too am dismayed at the abandonment of carve-reality-at-its-joints.

I'm surprised you think this is "absolutely critical". Do you think I'm making a grave error in my newfound distaste for paraphyletic groupings? (My ability to notice their awkwardness felt internally like evidence that my joint-carving skills have improved over the years, ftr.) Is there some other joint-carving skill you believe I am lacking, or have lost? Or perhaps you're decrying a decay in general community epistemics, for which my thread is simply a poster-child? Or perhaps you're lamenting some general community or global decline of candor? I'm uncertain what precisely you're dismayed about, and solicit specific criticisms of my actions.
I am not super creeped out by this. The example of dolphins didn't seem central to the general dynamics what the claim was about. "nitwit games" is a description of a logic you can't follow that you want to mock. The tree article made sense to me how a dolphin-fish carving would work. One of the virtues is to say "oops" and go on, this does make the mistakes more "stealthy". I don't like the bashing of ways of thinking so spewing hatred that way or this way is pretty passee anyway. One can be consistent in being against random fences and being for principled fences and call for a fencde to be taken down. But later learn that there was actually a principle and functioning of laying it down. Then it becomes an issue whether the original principles were better or worse than the new proposed principles. How long must one search before it starts to bereaosnable to treat the fence as purposeless? I think I would welcome reflection on how being mocky by surface examination is/was a bit misleading. There surely are a lot of biases and illusions but human brains try so assuming basic competence would give a somewhat favorable prior that a lot of words have some sensible structure in them. That some of them are pretty transparent and some are pretty obscure but that doesn't make it constructive to judge all obscure things by their apperance.

Meta note: I like how this is written. It's much shorter and more concise than a lot of the other posts you wrote in this sequence.

While the sort of Zettelkasten-adjacent notes that I do in Roam have really helped how I do research, I'd say No to this article. The literal Zettelkasten method is adapted to a world without hypertext, which is why I describe [what everyone does in Roam] as Zettelkasten-adjacent instead of Zettelkasten proper.

This is not to knock this post, it's a good overview of the literal Zettelkasten method. But I don't think it should be included.

Tempted as I may become, I will be extra careful not to discuss politics except as it directly relates to Covid-19 or requires us to take precautions for our own safety.


I don't think an oops is necessary here in this case (beyond just not crossing the norm again), but this is still appreciated. Thank you.

I suspect a bug. I have no recollection of turning personal blog posts on, but I still see the tag on next to Latest. It's entirely possible that I forgot about this, but that doesn't sound like a thing I'd do.

(That said, just realizing I can set a personal blog post penalty of -25 is going to make LessWrong much more tolerable.)

We've maintained backwards compatibility with many past iterations of filtering out personal blogposts. So you might have sometime in the past checked one of the checkboxes we had on the frontpage for filtering out personal blogposts.

These coronavirus posts are otherwise an excellent community resource and you are making them less valuable.

While I understand that this was first written for your own personal blog and then republished here, I do not believe that the entire section on Trump is appropriate in a LessWrong context. Not just in terms of Politics is the Mind Killer over the contentious claims you make, but primarily over the assertion that you can make contentious claims and shut down discussion over them. This seems like a very serious norms violation regarding what LessWrong is about.

Shutting down comments is not shutting down all discussion. If you have a disagreement that you find needs to be heard you can still write your own post to voice it.  Top-level posts are more important then comments for LessWrong and the LessWrong policy is that to encourage people to make top-level post this comes with the right to dictate the comment policy freely. 
You're not exactly wrong, but OP does tell us people are being irrational in ways that you could use to get cold hard cash.
I sympathize, but I think it's better if we allow this kind of thing, generally, under the conditions 'we' require now. And, as other comments mention, you can discuss these things, even on LessWrong. And I think it should be fine to make a comment, e.g. on this post, linking to your own response to the 'forbidden' topics.

Yeah, it's easier to abstain talking about politics if the article doesn't do it first.

Otherwise, great article and thanks for the work you put into it!

I do think this concern is right and appropriate to raise. I didn't include that section lightly, but didn't feel like I had a choice given the situation. I did realize that there was a cost to doing it.

As habryka says, these are written for my personal blog, and reposted to LessWrong automatically. I am happy that the community gets use out of them, but they are not designed for the front page of LessWrong or its norms. They couldn't be anyway, because time sensitive stuff is not front page material.

I don't believe I made contentious claims on non-Covid t... (read more)

The norms are that you get to talk about whatever you want, including election stuff, on your personal blog (which this and basically all other Coronavirus posts are on). We might hide things from the frontpage and all-posts page completely if they seem to get out of hand. On personal blog, bringing up a topic but asking others not to talk about it, also seems totally fine to me. If you want to respond you can always create a new top-level post (though in either case people might downvote stuff). 

Which claims, specifically, do you have evidence against (of any weight)? The fact that people are making mouth noises that could be interpreted as disagreeing with the accuracy of a claim (ie, the claim is contentious) is not evidence against the claim. And such mouth noises should not discourage the distribution of true information.
Load More