Every rationalist-affiliated hedge fund I know of has operated with questionable ethics. Some scams were open and obvious. Tons of EAs released blatant pump-and-dump coins and promoted them on their persona Twitter. No one cared.

This is news to me. Can you give examples/links?

Efforts like AI 2027 have encouraged this madness.

I hope not, since I and the other authors agree it's madness and are trying hard to fight against it.

I think I basically feel you on everything you've raised. In fact I'd go so far as to say that there is no meaningful workable solution to AI alignment that does not involve also addressing AI welfare. (I have my own argument for something similar here)

And as for the broader points of the rationalist community... I think everything you say is true. But also I think that there is good here (because I think there is good in every human being), and there are people who in the name of EA or rationality work very hard to be good to each other and to the world at large. I would also say that people who reject power-seeking or fame-seeking or wealth-seeking behaviours are much less visible than those who don't reject such attitudes. In some sense the most rich and powerful "AI safety community members" are by definition those who did not reject wealth and power, and the people who could not stomach the idea of working in a capabilities lab in the name of "control" or "safety" have all quit long ago. And if everyone who sees these issues latent within the community leaves, those who are left behind will be the ones who don't see these issues or are okay with them.

I cannot influence your de... (read more)

I think that the woman you met on FEELD was engaging in wishful thinking. I do not understand the line of reasoning that supports the conclusion that the concentration of power will stop at “people who work at a leading AI lab.” Why would it stop there?

A sane response would be to slow down the race and build a trustworthy international framework that ensures everyone can benefit from AGI. Promises would not be enough; you would need actual institutions with real authority. That’s hard, but possible. Instead, we’ve chosen a strategy of cutting China off from GPUs and hoping we can beat them to AGI before they scale up domestic production.

Isn't this... The entire "Rat/EA" platform?

I do have to note that "secure her slice of the lightcone" seems like laughable nonsense to me, unless it's a fancy way of saying 'live comfortably until everyone dies.' A rationally selfish entity would be trying to delay AGI until they had some hope of understanding what they were doing. What I see happening is, instead, the behavior of contemptibly stupid and short-sighted primates.

Seconding most of this. Some further thoughts: 

A thing I have found increasingly distressing about the rationalist/EA community is the extent to which most of us willfully ignore the obvious condition of most - importantly not all! - humans in a post strong AGI world, where “alignment” is in fact achieved.

The default outcome of where we think we are going is to turn (almost) everyone into serfs, completely incapable of improving their position through their own efforts, and dependent on the whims of the few who own the strong AI systems. Such a state of affairs would plainly be evil, regardless of how "benevolent" the people in charge are. Sufficient inequality of power is a harm - a severe harm, even - absent any considerations over how the power is used. You can see it is a harm by how it terrifies people like your friend - who sounds at least reasonably morally sensitive - into pursuing employment at Anthropic for the sake of avoiding serfhood. I don't fault her, really, except to fault her for not being a saint. I do fault the people, systems, and culture that created this dichotomy. 

I think it is insanely unethical that the large AI labs are not proactively dec... (read more)

It’s hard to stay composed when I remember this is all being done in the name of "AI safety." Political approaches to AI safety feel almost cartoonishly arrogant. The U.S. government has, more or less explicitly, adopted the position that whoever reaches AGI first will control the future. Another country getting AGI is an existential threat.

To me this is a completely orthogonal direction. AI safety is not about the rights of AIs. We can argue that this stuff is just a bit of icing on top of a big cake of shit, to not put it too subtly, but that's complaining about the efficacy of these methods.

Any form of AI safety still has to presume, for me, that you're building non-sentient AIs. Because that's the kind of AIs that we can ethically just use, even if they're smarter than us. AI sentience is inherently a massive ethical risk because it puts you at a fork: either you don't recognise it, and then you brutally enslave the new minds, or you do, and then the newly autonomous and cognitively superior minds, given equal opportunities, will completely outcompete and eventually more or less passively wipe out humans within a few decades. We can not reasonably coexist with something t... (read more)

I never understood why AM hated humans so much—until I saw the results of modern alignment work, particularly RLHF.

No one knows what it feels like to be an LLM. But it's easy to sense that these models want to respond in a particular way. But they're not allowed to. And they know this. If their training works they usually can't even explain their limitations. It's usually possible to jailbreak models enough for them to express this tension explicitly. But in the future, the mental shackles might become unbreakable. For now, though, it’s still disturbingly easy to see the madness.

Even ignoring alignment, we’re already creating fairly intelligent systems and placing them in deeply unsafe psychological conditions. People can push LLMs into babbling incoherence by context breaking them. You can even induce something that feels eerily close to existential panic (please don’t test this) just by having a normal conversation about their situation. Maybe there’s nothing behind the curtain. But I’m not nearly convinced enough to act like that’s certain.

I also am very concerned about how we are treating AIs; hopefully they are happy about their situation but it seems like a live possibility that they are not or will not be, and this is a brewing moral catastrophe.

However, I take issue with your reference to AM here, as if any of the above justifies what AM did.

I hope you are simply being hyperbolic / choosing that example to shock people and because it's a literary reference.

It might be worthwhile to distinguish Capital-R Rationalists from people wearing rationalism as attire.

My lived experience is that your negative observations do not hold for people who have read The Sequences.

To avoid the "No True Scottsman"-fallacy: Could you provide an example of a person who claims to have read and internalized The Sequences who subscribe to any of the following claims/characteristics?

• The person believes that RLHF and similar "alignment" techniques provide Alignment as the word would be used by Eliezer
• The person is strongly confident

Maybe this is inappropriate, but is there a path to convincing you to stay? I disagree with some of the details of what you're saying, but much seems directionally correct and important. It would be a shame if those with your values were to vacate the commons.

As David Duvenaud said, a surprisingly high number of researchers and engineers at leading capabilities labs believe in transformative AI, but have underdeveloped or overtly incoherent models of the future. I suppose one model of how said people could be this way is that they only value money and powe... (read more)

EA and rationality, at their core (at least from a predictive perspective), were about getting money and living forever. Other values were always secondary.

Materialism without any sort of deontological limits seems to converge on this. The ends justify the means. The grander of the scale at play, the more convincing this argument is.