Quick Takes

In 2011, Phil Goetz wrote the post "What we're losing," which bemoaned the shift from applied rationality to theoretical rationality. Regardless of the era we're in, I believe LessWrong has gone too far in the other direction: To some loose approximation, every post is now about AI. Even assuming it's what you care about more than anything, even coming from a prior of "the highest-EV move I can make is to write and interact with posts on x-risk," you should heavily consider the benefits (both social and practical) of diversifying your intake and output. Having norms around posting non-AI-related writing would be a good thing even for your ability to judge a person's output around topics you care about more, like AI safety. When the entire discussion is indexed around one topic (or a narrow range of topics), it becomes very easy to adjust the way you're speaking and thinking to "fit" properly within the distribution of speakers around you. This allows for bad actors and poor thinkers to have an easier time hiding themselves within a group without revealing themselves as such. This is, obviously, bad: A poor thinker operating in good faith should have the opportunity to be corrected, and a community is far better off when it's able to be mindful of which people are acting in bad faith. This is sort of a plea: Please, write some silly fiction. Write posts about getting better at thinking. Write some posts to help people get better at taking more general action! There's more juice to be squeezed out of the lemon of rationality, and when you're feeling anxious and can't stop staring at the end of the world, lemon juice will at least compliment the mood you're in.

Now seems an especially low-cost time for AGI company employees to make statements in support of pauses. Over the last week, there has been an unprecedented level of support for a pause/slowdown from AGI companies. A few examples: * The Anthropic RSI blog post * OpenAI's "our plan" blog post * Roon (OpenAI) on Twitter * Jasmine Wang (OpenAI) on Twitter * Stephen McAleer (Anthropic) on Twitter

Epistemic status: wild speculation There's an interesting line in the old MIRI paper on Definability of Truth: "Tarski’s result on the undefinability of truth is in some sense an artifact of the infinite precision demanded by reasoning about complete certainty." This feels vaguely analogous here to the uncertainty principle in quantum mechanics, which also prevents infinite-precision knowledge. (Has anyone discussed this before?) The connection jumped to my attention because I've also been thinking lately about the concept of causation in physical and logical universes. Intuitively speaking, the concept of Pearl's do-operator feels pretty sensible in our physical universe because there's a bounded lightcone which an event can affect—so that when we hypothesize taking an action, we can mostly ignore the implications for things in the past, or for simultaneous actions by other agents. (See this old post by Yudkowsky for more.) FDT generalizes the do-operator to the realm of logic, which doesn't have either timelike or spacelike separation by default. There's no privileged direction such that some logical facts affect others but aren't affected by them. And there's no notion of "distance" between different logical facts—e.g. any two facts A and B can be combined to give a proof of A AND B. It's possible that we could ground some version of logical timelike and spacelike separation using concepts like "what order does some proof system generate these facts in?", "how hard is it to prove one from the other?" or "how much computation do you save by calculating them together vs separately?" But I don't currently see much promise in this, since it all seems so representation-dependent. An alternative idea (inspired by some of davidad's thinking) is that intelligent minds might create logical timelike and spacelike separation as a consequence of their decisions. Demski's conception of logical time (which was anticipated by Lacan!) is mostly motivated by the idea of agen

It's annoying that you can't talk to Fable about basic biology, but I think it's good that they actually took biorisk seriously here despite annoying their customers. I'm more annoyed about the AI research restrictions since it won't tell you if the code you want it to write is forbidden and will just secretly half-ass it.

Is there a moment in each gifted child's life when you know so much that everyone accuses you of making things up? My 8 years old daughter was trying to tell her classmates about tardigrades -- tiny animals that are so resilient they can survive extreme temperature, pressure, starvation, dehydration, radiation, air deprivation, even exposure to outer space. First her classmates accused her of believing AI hallucinations. (Yes, my daughter often asks AI for information. Yeah, we told her not to believe everything. So far, the AI was mostly reliable.) Then they called a teacher, and she said that those are probably made up animals from my daughter's stories. (She likes to invent various stories, sometimes she writes them down or records them, but mostly just tells other people.) I guess it does not increase credibility to mention that tardigrades are also called "water bears" in English, "slow ones" in Slovak, and "little turtles" in Czech. So my wife printed a short summary from Wikipedia and a few other pages, and gave it to her to show at school. I am curious whether it will convince anyone.

Update on Claude playing Pokémon: Fable beat Pokémon FireRed with a vision-only harness in just over 50 hours. That's an hour faster than a heavily-harnessed GPT 5.5 beat FireRed, and >6x faster than the 325 hours it took a lightly-harnessed Opus 4.7 to beat Pokémon Red. (For comparison, an average human would take about 30 hours for FireRed and 26 hours for Red.) Unfortunately no full stream has been provided, just two short, edited videos, one of which was quickly taken down. Some analysis of the videos can be found here - for the first time there's some reason for suspicion that previous runs made it into the training data, though there isn't enough info to be confident either way.

Lot's of people believe we will be using AI to help us solve the alignment problem. So I did some poking at Claude Fable -- which is so enormously powerful in other areas -- on a relatively basic practical alignment question: Is the persona selection or goal alignment strategy better? I did multiple round where i had fable explain and critique both and then decide which one it preferred. It chose persona selection as the better alignment strategy and gave arguments why. Full final answer: [...] I think this is actually not particularly good reasoning? It points out that "goals frame's flaw is structural unfalsifiability" -- seems to me that there is plenty of stuff that can be done to empirically see if models are increasingly goal directed, such as AI wellbeing explored how models steer the world and I took a swing at it too. It claims the goals "predicts the same observations whether it's right or wrong" -- no? I think you would expect the model to steer the world according to goals and that being measurable. [I asked fable then to clarify structurally unfalsifiable: "the theory's own internal logic guarantees that no accessible observation can discriminate between the theory being true and false." This seems false about goal alignment?] Before this it claims: "never been shown to describe any actual mind, biological or artificial: nobody has found a goal slot". I never mentioned a goal slot in that conversation, obviously some minds are goal directed? But it seems to have smuggled in a much harder claim that for the goal framing of alignment to be true there must be a clean goal slot? That would make it certainly easier to align the systems but it's not necessary for goal alignment to be the better approach. Then it makes claims about expected-utility maximizers -- obviously things generally want to generally have more utility? But it appears to me that the term EUM is now often used to sort of gesture at monomaniacal paperclip maximizers, that old-fashioned