All of iceman's Comments + Replies

The passage is fascinating because the conclusion looks so self-evidently wrong from our perspective. Agents with the same goals are in contention with each other? Agents with different goals get along? What!?

Is this actually wrong? It seems to be a more math flavored restatement of Girardian mimesis, and how mimesis minimizes distinction which causes rivalry and conflict.

I was going to write something saying "no actually we have the word genocide to describe the destruction of a peoples," but walked away because I didn't think that'd be a productive argument for either of us. But after sleeping on it, I want to respond to your other point:

I don't think the orthogonality thesis is true in humans (i.e. I think smarter humans tend to be more value aligned with me); and sometimes making non-value-aligned agents smarter is good for you (I'd rather play iterated prisoner's dilemma with someone smart enough to play tit-for-tat

This is kind of the point where I despair about LessWrong and the rationalist community.

While I agree that he did not call for nuclear first strikes on AI centers, he said:

If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

and

Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange

This Facebook post.

Yeah, see, my equivalent of making ominous noises about the Second Amendment is to hint vaguely that there are all these geneticists around, and gene sequencing is pretty cheap now, and there's this thing called CRISPR, and they can probably figure out how to make a flu virus that cures Borderer culture by excising whatever genes are correlated with that and adding genes correlated with greater intelligence. Not that I'm saying anyone should try something like that if a certain person became US President. Just saying, you know, somebod

Over the years roughly between 2015 and 2020 (though I might be off by a year or two), it seemed to me like numerous AI safety advocates were incredibly rude to LeCun, both online and in private communications.

I think this generalizes to more than LeCun. Screencaps of Yudkowsky's Genocide the Borderers Facebook post still circulated around right wing social media in response to mentions of him for years, which makes forming any large coalition rather difficult. Would you trust someone who posted that with power over your future if you were a Borderer or... (read more)

Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.

Widely, does this come as a surprise? I recall back to the GPT2 days where the 4chan and Twitter users of AIDungeon discovered various prompting techniques we use today. More access means more people trying more things, and this should already be our base case because of how open participation in open source has advanc... (read more)

I have a very strong bias about the actors involved, so instead I'll say:

Perhaps LessWrong 2.0 was a mistake and the site should have been left to go read only.

My recollection was that the hope was to get a diverse diaspora to post in one spot again. Instead of people posting on their own blogs and tumblrs, the intention was to shove everyone back into one room. But with a diverse diaspora, you can have local norms to a cluster of people. But now when everyone is trying to be crammed into one site, there is an incentive to fight over global norms and attempt to enforce them on others.

Meta-note related to the question: asking this question here, now, means you're answer will be filtered for people who stuck around with capital r Rationality and the current LessWrong denizens, not the historical ones who have left the community. But I think that most of the interesting answers you'd get are from people who aren't here at all or rarely engage with the site due to the cultural changes over the last decade.

OK, but we've been in that world where people have cried wolf too early at least since The Hacker Learns to Trust, where Connor doesn't release his GPT-2 sized model after talking to Buck.

There's already been a culture of advocating for high recall with no regards to precision for quite some time. We are already at the "no really guys, this time there's a wolf!" stage.

Right now, I wouldn't recommend trying either Replika or character.ai: they're both currently undergoing major censorship scandals. character.ai has censored their service hard, to the point where people are abandoning ship because the developers have implemented terrible filters in an attempt to clamp down on NSFW conversations, but this has negatively affected SFW chats. And Replika is currently being investigated by the Italian authorities, though we'll see what happens over the next week.

In addition to ChatGPT, both Replika and character.ai are driving... (read more)

Didn't read the spoiler and didn't guess until half way through "Nothing here is ground truth".

I suppose I didn't notice because I already pattern matched to "this is how academics and philosophers write". It felt slightly less obscurant than a Nick Land essay, though the topic/tone aren't a match to Land. Was that style deliberate on your part or was it the machine?

While it’s nice to have empirical testbeds for alignment research, I worry that companies using alignment to help train extremely conservative and inoffensive systems could lead to backlash against the idea of AI alignment itself.

On the margin, this is already happening.

Stability.ai delayed the release of Stable Diffusion 2.0 to retrain the entire system on a dataset filtered without any NSFW content. There was a pretty strong backlash against this and it seems to have caused a lot of people to move towards the idea that they have to train their own mod... (read more)

Zack's series of posts in late 2020/early 2021 were really important to me. They were a sort of return to form for LessWrong, focusing on the valuable parts.

What are the parts of The Sequences which are still valuable? Mainly, the parts that build on top of Korzybski's General Semantics and focus hard core on map-territory distinctions. This part is timeless and a large part of the value that you could get by (re)reading The Sequences today. Yudkowsky's credulity about results from the social sciences and his mind projection fallacying his own mental quirk... (read more)

The funny thing is that I had assumed the button was going to be buggy, though I was wrong how. The map header has improperly swallowed mouse scroll wheel events whenever it's shown; I had wondered if the button would also interpret them likewise since it was positioned in the same way, so I spent most of the day carefully dragging the scrollbar.

There must be some method to do something, legitimately and in good-faith, for people's own good.

"Must"? There "must" be? What physical law of the universe implies that there "must" be...?

Let's take the local Anglosphere cultural problem off the table. Let's ignore that in the United States, over the last 2.5 years, or ~10 years, or 21 years, or ~60 years (depending on where you want to place the inflection point), social trust has been shredded, policies justified under the banner of "the common good" have primarily been extractive and that in the US, ... (read more)

This seems mostly wrong? A large portion of the population seems to have freedom/resistance to being controlled as a core value, which makes sense because the outside view on being controlled is that it's almost always value pumping. "It's for your own good," is almost never true and people feel that in their bones and expect any attempt to value pump them to have a complicated verbal reason.

The entire space of paternalistic ideas is just not viable, even if limited just to US society. And once you get to anarchistic international relations...

I agree that paternalism without buy-in is a problem, but I would note LessWrong has historically been in favor of that: Bostrom has weakly advocated for a totalitarian surveillance state for safety reasons and Yudkowsky is still pointing towards a Pivotal Act which takes full control of the future of the light cone. Which I think is why Yudkowsky dances around what the Pivotal Act would be instead: it's the ultimate paternalism without buy-in and would (rationally!) cause everyone to ally against it.

What changed with the transformer? To some extent, the transformer is really a "smarter" or "better" architecture than the older RNNs. If you do a head-to-head comparison with the same training data, the RNNs do worse.

But also, it's feasible to scale transformers much bigger than we could scale the RNNs. You don't see RNNs as big as GPT-2 or GPT-3 simply because it would take too much compute to train them.

You might be interested in looking at the progress being made on the RWKV-LM architecture, if you aren't following it. It's an attempt to train an RNN like a transformer. Initial numbers look pretty good.

I think the how-to-behave themes of the LessWrong Sequences are at best "often wrong but sometimes motivationally helpful because of how they inspire people to think as individuals and try to help the world", and at worst "inspiring of toxic relationships and civilizational disintegration."

I broadly agree with this. I stopped referring people to the Sequences because of it.

One other possible lens to filter a better Sequences: is it a piece relying on Yudkowsky citing current psychology at the time? He was way too credulous, when the correct amount to up... (read more)

I want to summarize what's happened from the point of view of a long time MIRI donor and supporter:

My primary takeaway of the original post was that MIRI/CFAR had cultish social dynamics, that this lead to the spread of short term AI timelines in excess of the evidence, and that voices such as Vassar's were marginalized (because listening to other arguments would cause them to "downvote Eliezer in his head"). The actual important parts of this whole story are a) the rationalistic health of these organizations, b) the (possibly improper) memetic spread of t... (read more)

That sort of thinking is why we're where we are right now.

Be the change you wish to see in the world.

I have no idea how that cashes out game theoretically. There is a difference between moving from the mutual cooperation square to one of the exploitation squares, and moving from an exploitation square to mutual defection. The first defection is worse because it breaks the equilibrium, while the defection in response is a defensive play.

swarriner's post, including the tone, is True and Necessary.

It's just plain wrong that we have to live in an adversarial communicative environment where we can't just take claims at face value without considering political-tribe-maneuvering implications.

Oh? Why is it wrong and what prevents you from ending up in this equilibrium in the presence of defectors?

More generally, I have ended up thinking people play zero-sum status games because they enjoy playing zero-sum status games; evolution would make us enjoy that. This would imply that coordination beats epistemics, and historically that's been true.

[The comment this was a response to has disappeared and left this orphaned? Leaving my reply up.]

But there's no reason to believe that it would work out like this. He presents no argument for the above, just pure moral platitudes. It seems like a pure fantasy.

As I pointed out in the essay, if I were running one of the organizations accepting those donations and offering those prizes, I would selectively list only those targets who I am genuinely satisfied are guilty of the violation of the "non-aggression principle." But as a practical matter, there is n

you just need to find the experts they're anchoring on.

I believe we are in the place we are in because Musk is listening and considering the arguments of experts. Contra Yudkowsky, there is no Correct Contrarian Cluster: while Yudkowsky and Bostrom make a bunch of good and convincing arguments about the dangers of AI and the alignment problem and even shorter timelines, I've always found any discussion of human values or psychology or even how coordination works to be one giant missing mood.

(Here's a tangential but recent example: Yudkowsky wrote his De... (read more)

"well, I sure have to at least do a lot of straussian reading if I want to understand what people actually believe, and should expect that depending on the circumstances community leaders might make up sophisticated stories for why pretty obviously true things are false in order to not have to deal with complicated political issues"

I kinda disagree that this is a mere issue of Straussian reading: I suspect that in this (and other cases), you are seeing the raw output of Elizer's rationalizations and not some sort of instrumental coalition politics dark ... (read more)

We must all remember essential truths: that while we write about Clippy, we do that because Clippy is an amusing fiction. In reality, Clippy is significantly less likely to come into existence than CelestAI. An AI being trained is more likely to be CelestAI than a generic paperclipper. CelestAI is more probable. The probability of discussion of paperclips on the internet is less likely than discussion about My Little Pony: Friendship is Magic. One could query popular language models to verify this. More people would try to build CelestAI than an AI to maxi... (read more)

n=1, but I have an immediate squick reaction to needles. Once vaccines were available, I appeared to procrastinate more than the average LWer about getting my shots, and had the same nervous-fear during the run up to getting the shot that I've always had. I forced myself through it because COVID, but I don't think I would have bothered for a lesser virus, especially at my age group.

Isn't this Moldbug's argument in the Moldbug/Hanson futarchy debate?

(Though I'd suggest that Moldbug would go further and argue that the overwhelming majority of situations where we'd like to have a prediction market are ones where it's in the best interest of people to influence the outcome.)

While I vaguely agree with you, this goes directly against local opinion. Eliezer tweeted about Elon Musk's founding of OpenAI, saying that OpenAI's desire for everyone to have AI has trashed the possibility of alignment in time.

FYI, there's a lot of links that don't work here. "multilevel boxing," "AI-nanny," "Human values," and so on.

The only reward a user gets for having tons of karma is that their votes are worth a bit more

The only formal reward. A number going up is its own reward to most people. This causes content to tend closer to consensus: content people write becomes a Keynesian beauty contest over how they think people will vote. If you think that Preference Falsification is one of the major issues of our time, this is obviously bad.

why do you think it is a relevant problem on LW?

I mentioned the Eugene Nier case, where a person did Extreme Botting to manipulate the scores of people he didn't like, which drove away a bunch of posters. (The second was redacted for a reason.)

After this and the previous experiments on jessicata's top level posts, I'd like to propose that these experiments aren't actually addressing the problems with the karma system: the easiest way to get a lot of karma on LessWrong is to post a bunch (instead of working on something alignment related), and the aggregate data is kinda meaningless and adding more axis doesn't fix this. The first point is discussed at length on basically all sites that use upvote/downvotes (here's one random example from reddit I pulled from Evernote), but the second isn't. Give... (read more)

In wake of the censorship regime that AI Dungeon implemented on OpenAI's request, most people moved to NovelAI, HoloAI, or the open source KoboldAI run on colab or locally. I've set up KoboldAI locally and while it's not as featureful as the others, this incident is another example of why you need to run code locally and not rely on SaaS.

For background, you could read 4chan /vg/'s /aids/ FAQ ("AI Dynamic Storytelling"). For a play-by-play of Latitude and OpenAI screwing things up, Remember what they took from you has the history of them leaking people's personal stories to a 3rd party platform.

somewhere where you trust the moderation team

That would be individual's own blogs. I'm at the point now where I don't really trust any centralized moderation team. I've watched some form of the principal agent problem happen with moderation repeatedly in most communities I've been a part of.

I think the centralization of LessWrong was one of many mistakes the rationalist community made.

Assuming that language is about coordination instead of object level world modeling, why should we be surprised that there's little correlation between these two very different things?

My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.

Google's a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.

I want to second this. I worked for an organization where one of key support people took psychedelics and just...broke from reality. This was both a personal crisis for him and an organizational crisis for the company to deal with the sudden departure of a bus factor 1 employee.

I suspect that psychedelic damage happens more often than we think because there's a whole lobby which buys the expand-your-mind narrative.

I can verify that I saw some version of their document The Plan[1] (linked in the EA Forum post below) in either 2018 or 2019 while discussing Leverage IRL with someone rationalist adjacent that I don't want to doxx. While I don't have first hand knowledge (so you might want to treat this as hearsay), my interlocutor did and told me that they believed they were the only one with a workable plan, along with the veneration of Geoff.

[1]: I don't remember all of the exact details, but I do remember the shape of the flowchart and that looks like it. It's possib... (read more)

Am I the only one creeped out by this?

Usually I don't think short comments of agreement really contribute to conversations, but this is actually critical and in the interest of trying to get a public preference cascade going: No. You are not the only one creeped out by this. The parts of The Sequences which have held up the best over the last decade are the refinements on General Semantics, and I too am dismayed at the abandonment of carve-reality-at-its-joints.

Meta note: I like how this is written. It's much shorter and more concise than a lot of the other posts you wrote in this sequence.

While the sort of Zettelkasten-adjacent notes that I do in Roam have really helped how I do research, I'd say No to this article. The literal Zettelkasten method is adapted to a world without hypertext, which is why I describe [what everyone does in Roam] as Zettelkasten-adjacent instead of Zettelkasten proper.

This is not to knock this post, it's a good overview of the literal Zettelkasten method. But I don't think it should be included.

Tempted as I may become, I will be extra careful not to discuss politics except as it directly relates to Covid-19 or requires us to take precautions for our own safety.

I don't think an oops is necessary here in this case (beyond just not crossing the norm again), but this is still appreciated. Thank you.

I suspect a bug. I have no recollection of turning personal blog posts on, but I still see the tag on next to Latest. It's entirely possible that I forgot about this, but that doesn't sound like a thing I'd do.

(That said, just realizing I can set a personal blog post penalty of -25 is going to make LessWrong much more tolerable.)

These coronavirus posts are otherwise an excellent community resource and you are making them less valuable.

While I understand that this was first written for your own personal blog and then republished here, I do not believe that the entire section on Trump is appropriate in a LessWrong context. Not just in terms of Politics is the Mind Killer over the contentious claims you make, but primarily over the assertion that you can make contentious claims and shut down discussion over them. This seems like a very serious norms violation regarding what LessWrong is about.