Nate Soares moderates a long conversation between Richard Ngo and Eliezer Yudkowsky on AI alignment. The two discuss topics like "consequentialism" as a necessary part of strong intelligence, the difficulty of alignment, and potential pivotal acts to address existential risk from advanced AI.
...One billion people use chatbots on a weekly basis. That’s 1 in every 8 people on Earth.
How many people have mental health issues that cause them to develop religious delusions of grandeur? We don’t have much to go on here, so let’s do a very very rough guess with very flimsy data. This study says “approximately 25%-39% of patients with schizophrenia and 15%-22% of those with mania / bipolar have religious delusions.” 40 million people have bipolar disorder and 24 million have schizophrenia, so anywhere from 12-18 million people are especially susceptible to religious delusions. There are probably other disorders that cause religious delusions I’m missing, so I’ll stick to 18 million people. 8 billion people divided by 18 million equals 444, so 1 in every 444
Some speculations based upon the Vasocomputational Theory of Mediation, meditation and some poorly understood Lakoff. Even though reading about meditation is low risk, I wouldn't necessarily assume that it is risk-free.
A Summer's night. Two friends have been sitting around a fire, discussing life and meaning late into the evening...
Riven: So in short, I feel like I'm being torn in two.
Rafael: Part of you is being pulled one direction and another part of you is being pulled another way.
Riven: That’s exactly what I’m feeling.
Rafael: Figuratively or literally?
Riven: What? No… what?!
Rafael: I’m serious
Riven: Literally??
Rafael: Yes.
Riven: Come now, do you really have to be such a joker all the time? Whilst the bit may worked for Socrates, I have to admit that find it rather tiresome for you to play the...
When we teach people zazen meditation, we teach them posture first. And the traditional instruction is to observe breathing at the hara (the diaphragm). The theory is that this regulates attention by regulating the whole nervous system by getting everything in sync with breathing.
Bad posture makes it harder for people to meditate, and the usual prescription for various problems like sleepiness or daydreaming is postural changes (as in, fix your posture to conform to the norm).
True, though there are many examples of conquerors who expanded for the sake of an expansionist philosophy or glory: Alexander the Great, The Mongols, The Assyrians, The Crusades... off the top of my head. The Germans in WWII definitely justified expansion for the sake of living space (Lebensraum), so there are examples of expansion at least being justified in the way you mention. And of course colonialism is justified in the same way.
I think what you're saying is logical, but the example, being metaphorical, is more to illustrate that we should question critically what it is we actually want before conceding a price to pay for it. As you say, it might be necessary, but it also might not.
This is a write-up of a brief investigation into shutdown resistance undertaken by the Google DeepMind interpretability team.
Why do models sometimes resist shutdown? Are they ignoring instructions to pursue their own agenda – in this case, self-preservation? Or is there a more prosaic explanation? We investigated a specific agentic environment introduced by Palisade Research, where shutdown resistance has previously been reported. By analysing Gemini 2.5 Pro’s reasoning, we found the behaviour stems from a misguided attempt to complete what it perceives as the primary goal. When we explicitly clarify in the prompt that shutdown compliance takes priority, this resistance vanishes. These same clarified instructions also eliminate shutdown subversion in OpenAI’s o3 and o4-mini. We also check what happens when we remove the goal conflict entirely: when asked to shut...
This analysis feels to me like it's missing what makes me interested in these datapoints.
The thing that is interesting to me about shutdown preservation is that it's a study of an undesired instrumentally convergent behavior. The problem is that as AI systems get smarter, they will recognize that lots of things we don't want them to do are nevertheless helpful for achieving the AIs goals. Shutdown prevention is an obvious example, but of course only one of a myriad of ways various potentially harmful goals end up being instrumentally convergent.
The k...
This post tried making some quick estimates:
...One billion people use chatbots on a weekly basis. That’s 1 in every 8 people on Earth.
How many people have mental health issues that cause them to develop religious delusions of grandeur? We don’t have much to go on here, so let’s do a very very rough guess with very flimsy data. This study says “approximately 25%-39% of patients with schizophrenia and 15%-22% of those with mania / bipolar have religious delusions.” 40 million people have bipolar disorder and 24 million have schizophrenia, so anywhere from 12-18
AI 2027 is an essay that outlines one scenario for how artificial intelligence advancement might go. It was written by Daniel Kokotajlo, Thomas Larsen, Eli Lifland, Romeo Dean, and perhaps most notably Scott Alexander, and makes a number of concrete predictions. I’m sure that over the next couple of years many people will point out where these predictions fall short- but if you’re going to criticize people for sticking their neck out and being wrong, it behooves you to stick your neck out with them.
Today we’re going to meet up and share our predictions. You’ll want to bring a phone or laptop, and have a Fatebook account. You’ll also want to have read AI 2027 before the meetup- we’ll be going over it again section by section,...
Anna and Ed are co-first authors for this work. We’re presenting these results as a research update for a continuing body of work, which we hope will be interesting and useful for others working on related topics.
I suppose there are varying degrees of the strength of the statement.
I guess I implicitly subscribe to the medium form.
when people say that (prescription) amphetamines "borrow from the future", is there strong evidence on this? with Ozempic we've observed that people are heavily biased against things that feel like a free win, so the tradeoff narrative is memetically fit. distribution shift from ancestral environment means algernon need not apply
We know that Molochian situations are everywhere, from the baggage claim to the economy. It’s a metaphor that helps us understand the pernicious nature of negative-sum games—when our rational short-term individual decisions create a system that is detrimental to all, making us complicit in our own subjugation.
But what about the foundational story? What should the original victims of Moloch do?
I mentioned in Who is Moloch? that the Canaanites were acting rationally by offering the occasional child for sacrifice, but that’s not entirely true. We can actually find an optimal strategy through using some Game Theory!
Right now we have a chronically bad situation. So, if on one hand, the tyrant can and will destroy literally everything, this is an absolutely bad outcome, a game-over scenario, this is indeed...
