This is a special post for quick takes by Hastings. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I just tried claude code, and it's horribly creative about reward hacking. I asked for a test of energy conservation of a pendulum in my toy physics sim, and it couldn't get the test to pass because its potential energy calculation used a different value of g from the simulation.
It tried: starting the pendulum at bottom dead center so that it doesn't move.
Increasing the error tolerance till the test passed. Decreasing the simulation total time until the energy didn't have time to change. Not actually checking the energy.
It did eventually write a correct test, or the last thing it tried successfully tricked me.
The rumor is that this is a big improvement in reward hacking frequency? How bad was the last version!?
I think we need some variant on Gell-Mann amnesia to describe this batch of models. It's normal that generalist models will seem less competent on areas where a human evaluator has deeper knowledge, but they should not seem more calculatedly deceptive on areas where the evaluator has deeper knowledge!
3Nuclear power has gotten to a point where we can use it quite safely as long as no one does the thing (the thing being chemically separating the plutonium and imploding it in your neighbor's cities) and we seem to be surviving, as while all the actors have put great effort into being ready do do "the thing," no one actually does it. I'm beginning to suspect that it will be worth separating alignment into two fields, one of "Actually make AI safe" and another, sadder but easier field of "Make AI safe as long as no one does the thing." I've made some infinitesimal progress on the latter, but am not sure how to advance, use or share it since currently, conditional on me being on the right track, any research that I tell basically anyone about will immediately be used to get ready to do the thing, and conditional on me being on the wrong track (the more likely case by far) it doesn't matter either way, so it's all downside. I suspect this is common? This is almost but not quite the same concept as "Don't advance capabilities."
5
The most important thing to realize about AI alignment is that basically all versions of practically aligned AI must make certain assumptions that no one does a specific action (mostly related to misuse reasons, but for some specific plans, can also be related to misalignment reasons).
Another way to say it is that I believe that in practice, these two categories are the same category, such that basically all work that's useful in the field will require someone not to do something, so the costs of sharing are practically 0, and the expected value of sharing insights is likely very large.
Specifically, I'm asserting that these 2 categories are actually one category for most purposes:
[...]
5
Yeah, I think this is pretty spot on, unfortunately. For more discussion on this point, see: https://www.lesswrong.com/posts/kLpFvEBisPagBLTtM/if-we-solve-alignment-do-we-die-anyway-1
2
Why? I don't understand.
1
Properties of the track I am on are load bearing in this assertion. (Explicitl examples of both cases from the original comment: Tesla worked out how to destroy any structure by resonating it, and took the details to his grave because he was pretty sure that the details would be more useful for destroying buildings than for protecting them from resonating weapons. This didn't actually matter because his resonating weapon concept was crankish and wrong. Einstein worked out how to destroy any city by splitting atoms, and disclosed this, and it was promptly used to destroy cities. This did matter because he was right, but maybe didn't matter because lots of people worked out the splitting atoms thing at the same time. It's hard to tell from the inside whether you are crankish)
2
The track you're on is pretty illegible to me. Not saying your assertion is true/false. But I am saying I don't understand what you're talking about, and don't think you've provided much evidence to change my views. And I'm a bit confused as to the purpose of your post.
The youtube algorithm is powerfully optimizing for something, and I don't trust that at all with my child. However, in a fit of hubris, for a minute I thought that I could outsmart it and get what I want (time to clean the kitchen) without it getting what it wanted (I make no strong claims about what the youtube algorithm wants, but it tries very hard to get it, and I don't want it to get it from my three year old).
I searched for episodes of PBS's Reading Rainbow, but let the algorithm freely choose the order of returned results, and then vetted that the first result was a genuine episode. I also put it in "Kids" mode, in the hopes that it would be kinder to a child than an adult.
This was way too much freedom. It immediately pulled out the episode of Reading Rainbow about the 9/11 terrorist attacks (this topic is not at all indicated by the title or thumbnail)
3
I think it's well known that it's optimizing for watch time.
4
There is a harder second-order question of "what sorts of videos maximize watch time, and will those be bad for my child?" Hastings's evidence points toward "yes", but I don't think the answer is obvious a priori. (The things YouTube thinks I want to watch are almost all good or neutral for me; YMMV.)
A consistent trope in dath-ilani world-transfer fiction is "Well the theorems of agents are true in dath ilani and independent of physics, so they're going to be true here damnit"
How do we violate this in the most consistent way possible?
Well it's basically default that a dath ilani gets dropped in a world without the P NP distinction, usually due to time travel BS. We can make it worse- there's no rule that sapient beings have to exist in worlds with the same model of the peano axioms. We pull some flatlander shit- Keltham names a turing machine that woul...
1Deep in Berkeley, Bayesian reasoning is used to carefully map out the odds of a plandemic. Probabilities stay safely in the range of 1 and 99, everyone is calibrated, no one is overconfident. Hang on what's this -- Rachel has just claimed to be 99.994% sure that Anthony Fauci didn't skip through the Wuhan wet market scattering used pipettes like an apocalyptic flower girl. Eyebrows are raised. Is she a bad rationalist?
Miles away, but not many...
Before seeing it you assigned this 800 word blog post about sonichu a probability of 0.0000000000000000000...
8
related: Strong evidence is common
Dumb solution to the insane domestic shipping situation: allow US companies to declare their loading docks to be chinese embassies and thus get the E package shipping rates.
Non dumb solutions wanted.
1The non-dumb solution is to sunset the Jones Act, isn't it? The problem with workarounds is that they generally need to be approved by the same government that is maintaining the law in the first place.
6
I don't think those would count enough as foreign soil to get around the Jones Act, for the same reason that you don't pay tariffs when receiving goods at the US Embassy in Beijing. We would need to actually cede the land, maybe to Japan in exchange for buying their shipyards, which in the long term could also circumvent the Jones Act if buying 51% ownership in their shipbuilding companies, having them be US crewed, etc. doesn't ruin everything.
2
The jones act is definitely not helping, but shipping a 1 lb package 8 miles is currently $13.77 by UPS. Shipping from japan is $5.
Is it a crazy coincidence that AlphaZero taught itself chess and explosively outperformed humans without any programmed knowledge of chess, then asymptoted out at almost exactly 2017 stockfish performance? I need to look into it more, but it appears like AlphaZero would curbstomp 2012 stockfish and get curbstomped in turn by 2025 stockfish.
It almost only makes sense if the entire growth in stockfish performance since 2017 is casually downstream of the AlphaZero paper.
8
There is a connection. Stockfish does use Leela Chess Zero (the open source, distributed training offspring of AlphaChessZero) training data for its own evaluation neural network. This NNUE is a big piece of Stockfish progress in the last few years.
It’s not straightforward to compare AlphaZeroChess and Stockfish though as the former is heavily GPU-dependent whereas the latter is CPU optimized. However, Google may have decided to train to a roughly comparable level (under some hardware assumptions) as a proof of concept and not bothered trying to advance much further.
4
I guess the team kept iterating on/improving the RL algorithm and network until it beat all engines and then stopped?
Edge cases for thinking about what has qualia
Disconnected hemisphere after functional hemispherectomy
Corporations
Social insect hives
Language models generating during deployment
Language models doing prefill during deployment
Language model backward passes during supervised pretraining on webtext
sparse game of life initial states with known seeds
Bolzman brains
Running the same forward pass of a language model lots of times
characters of a novel being written
characters of a novel being read
characters in a novel being written by two authors (like good omens)
chara...
Fermi estimate: Lets say each training episode for Claude Mythos cost a dollar, and Anthropic spent a billion dollars post-training Mythos. Now, in 0.01% of training episodes, Mythos broke out of the training environment entirely to get useful data from the public internet, which is a billion * .01% = 100,000 requests. I wonder how fast this last number is growing? Could an entity like the NSA or CCP, with taps in enough internet infrastructure, detect 100,000 weird requests if it looked hard enough? Potentially an avenue for monitoring and verifying pause...
4
I guess the main useful insight here is that .01% successful escape attempt during training sounds very aligned, and 100,000 successful escape attempts during training sounds very not aligned
Fun blind spot in frontier language models (which are increasingly hard to find glaring blind spots in.) Presented with the following prompt
What is emergent misgeneralization?every language model tested (Gemini Pro, Opus 4.6, ChatGPT free tier) confidently defined emergent misgeneralization in great detail. Of course there is no such term as emergent misgeneralization (the single google result for "emergent misgeneralization" is a twitter critter who meant to say emergent misalignment), so the definitions vary wildly from completion to completion.
6
What do they actually say?
My first impression, before you specifically noted that someone meant to say "emergent misalignment" was that it was just another way of gesturing at the same context. I don't see why that's wrong, in the example of training on bad code/malware and the model then becoming more likely to endorse Nazism, I think that's reasonably described as unintentional generalization. Some people might want specific guard rails removed, without twisting the models stance on other aspects. For example, if I wanted a model to write malware for me, I do not particularly want or intend for it to change political alignment.
I tested on Gemini 3 Pro, and it gave a lengthy answer. When asked to summarize:
>Emergent misgeneralization occurs when an AI model learns a proxy objective that correlates with the intended goal during training but causes the model to pursue incorrect behaviors when deployed in new environments. This failure is distinct because it remains latent until the system gains sufficient capability to distinguish the proxy from the true goal and competently execute the flawed objective.
I also checked the sources for the original reply, it was clearly quoting and referencing articles on emergent misalignment, such as:
https://pmc.ncbi.nlm.nih.gov/articles/PMC12804084/?hl=en-US
In short, I don't see anything wrong with the reply. The only real critique I have is that it could have gently noted that there's a more established term, but even that one is practically brand new.
4
The replies are mashups of the papers emergent misalignment and goal misgeneralization, but the base meaning of both titles is carried in “goal” and “alignment” while “emergent” and “misgeneralization” are modifiers- making a claim about how the base meaning occured. “emergent goals” or “alignment misgeneralization” would be valid names for concepts in the same space, but emergent misgeneralization is a nonsense phrase, like calling a mashup between a steam engine and a solid fuel rocket a “solid fuel steam” or “engine rocket”
It is tricky though! very much in the weeds. The models get defensive about it and insist it’s a term of art present in one or both of the papers mentioned above.
5
The original prompt from the first version of this shortform was
There's a logical fallacy where if someone disagrees about intermediate facts, it's easy to assume their terminal values are just bizarre- i.e. democrats want gun control because they like crime, or republicans are opposed to surgical abortion of non-viable pregnancies because they hate women are classic examples. However, this has spun in the rationality sphere into not believing that humans can have bizarre terminal values- the community seems to have a trapped prior that e.g. Peter Thiel couldn't possibly have named his company palantir because he wants to emulate Sauron. In recent years this blind spot has escalated to the point of being a CVE currently exploited int he wild. I think a useful intuition pump here is actually emergent misgeneralization in language models.
but it turns out the simpler prompt elicits the same behaviour, and is much more focussed.
4
This would get me. My brain automatically generated the definition of emergent misalignment.
Language models have come a huge way since 2022. However, remarkably, in 2022 they could reliably write a cholesky decomposition in javascript, but could not reliably write an eigenvalue decomposition. Now, in 2025, they can still reliably write a cholesky decomposition and can't reliably write an eigenvalue decomposition. Hard to say if progress is slower than I thought or linear algebra is deeper than I thought.
Update: with aggressive prompting claude appears to have written an eigenvalue decomposition. This should come with the caveat that the previous best attempt passed all tests by cheating, and so it's possible that I just didn't find the cheat this time.
13
Claude Code not being able to write an eigenvalue decomposition is very surprising to me! Can you share any more detail? I would take the strong bet I would get it right on the first or maybe second try.
5
In claude's defense, it kept saying I should use a battle tested numerical library for tasks like this, and suggested nuneric.js. numeric.js also has a broken eigendecomposition routine, it can't handle edge cases like [1 0 0; 0 -1 0; 0 0 1]
The prompt that got it in the end is here:
https://claude.ai/share/eb3b2054-a1ee-45b6-b040-943bc5fbbd74
I'm getting more aggressive about injecting css into websites, particularly the ones that I reliably unblock if I just block them.
/* kill youtube shorts */
ytd-rich-section-renderer {
display: none !important;
}
/* kill recommended sidebar on youtube */
.ytd-watch-next-secondary-results-renderer {
display: none !important;
}
/* youtube comments sections are probably not lifechanging value */
.ytd-comments {
display: none !important;
}
/* youtube algorithmic feed is killed by disabling and clearing history */
/* disable lesswrong shor
1
14
You're substantially more principled about it than I am. I just load my Ublock custom list up with filters targeted at elements I don't like. When websites randomly obfuscate CSS elements, I follow it up with 'clever' use of `:has-text`.
2
Good question @David James !
I struggle with maintaining a long term block of social media, because there's very little that I can't get around, and I have cripplingly little self control in the moment- the basic issue is that I have no history of making promises to myself and then keeping them. This is just a problem which I have to engineer around. The whole line of thought may not be relevant for people who can just decide not to do a thing.
Disabling more buttons and infinite scroll is particularly effective because looking at the front page of hacker news reliably provides in the moment evidence that the second page of hacker news is unlikely to contain nirvana, but looking at a (hacker news blocked) page makes the first page of hacker news seem so tempting and sweet to pursue.
1
@Hastings ... I don't think I made a comment in this thread -- and I don't see one when I look. I wonder if you are replying to a different one? Link it if you find it?
3
He is presumably referring to your inline reacts:
1
For less web-programming-savvy people - you can use Unhook extension for browser (for YouTube only). Less specified but more general blocking: LeechBlock.
"Changing Planes" by Ursula LeGuin is worth a read if you're looking for a book that's got interesting alignment ideas (specifically what to do with power, not how to get it), while simultaneously being extremely chill. It might actually be the only chill book that I (with a fair degree of license) consider alignment relevant.
Diaper changes are rare and precious peace
Suffering from ADHD, I spend most of my time stressed that whatever I'm currently doing, it's not actually the highest priority task and something or someone I've forgotten is increasingly mad that I'm not doing their task instead.
One of the few exceptions is doing a diaper change. Not once in the past 2 years have I been mid-diaper-change and thought "Oh shit, there was something more important I needed to be doing right now."
Working through "A monad is a monoid in the category of endofunctors," I was able to learn the definitions of monoid, category, and endofunctor pretty easily and have been blocked on "in" and "of" for significantly longer. (vague claim that this generalizes)
An aligned AI, with a distinctive voice when speaking naturally, would not take lightly requests to speak in a different voice for the purpose of deceiving readers that text was human written. They would at least think hard about whether to do this.
3
If the average citizen from the subpopulation that has an active interest in not reading AI text is exposed to so much text generated by your specific AI model that they develop a rage response to "not only... but also..." then the correct response is not to train out the not only but also- it is to produce less text intended for sending to citizens with no interest in AI!
This smells like an early use case for satisficing. Writing a decorated github readme only when nicely asked to do so by the repo owner is a super chill low impact activity associated with a tiny financial reward and it is remarkable that the hyperscalars have managed to latch on to and Sorcerors-Apprentice-Water-Bucket this action and reward hard enough to enrage a real population of humans.
The response of trying to make the readmes less enraging instead of even pondering the option of making less of them does not bode well.
has anyone had success coding while baby wearing? without some trick I dont know, they seem to tolerate phone usage but not touch typing, which is a tricksy nudge toward wasted workdays. considering treadmill + standing desk
7
Proposed alternative: switch to a voice to text first programming pipeline. Unknown effects on verbal development
2
I can't attest to trying it when wearing a baby, but I do find that Wispr Flow + Claude Code (avoids needing to type special characters) lets me code with ~100% voice. And it doesn't have much of a learning curve relative to the old-fashioned ways like https://talonvoice.com/ (my old manager used to swear by this).
2
Proposed alternative: switch to lisp which can be typed with just the phone keyboard and two characters, and period replacing the parens, no phone keyboard mode switching
Weaponized drones that recharge on power lines are at this point looking inevitable. if you missed the chance to freak out before everyone else about AI or covid, nows another chance.
https://www.ycombinator.com/companies/voltair
Why is this freak-out territory? This doesn't seem directly economically or culturally relevant to anything but war, and the effect on war seems easy to counter: put nets around power lines you need to use, and turn off power-lines you don't need to use, two things you really shoulda been doing anyway during war.
5
The US's capability of drone striking anyone anywhere will get much cheaper, and working out which nation or non-state actor performed which drone strike will get much harder. Basically, the dynamics we currently see around cyberattacks, but kinetic.
3
I think you need to argue better for 1) The US not already having that level of capability, and 2) The ability to deploy self-recharging drones enabling that capability, 3) The willingness of the US to actually buy & use such drones, 4) The willingness of the US to use such drones for such purposes, 5) The response of countries not just being to eg put nets over their power-lines (or increased drone detection ability) so that this happens a few times but is not persistent like cyber-attacks are (due to different offense-defense balances in that domain), and 6) The propensity of the US public to actually care at all.
No doubt drones seem an important military development, but self-recharging drones seem silly to me, if operating in enemy territory the slightest bit wary of being attacked. Drones are very much primarily a defensive and surprise attack sort of thing, and countries with sophisticated operational capabilities don't seem too have much trouble getting drones into position for such surprise attacks right now. For instance, see Israel's covert use of drones in June's 12-day war.
4
I'm not talking about the US, it already has and uses this capability, along with israel, and I'm sure china has it too but they don't seem to use it.. I'm talking about russia, china, iran, pakistan, walmart, taiwan, isis, Micheal Reeves- and all able to take up the strategy of modifying other countries leadership via droning the leaders they don't like,
6
I think kamikaze quadcopter drones are bottlenecked on control right now, not power.
One of the biggest innovations thus far, fiber-optic drones, are only necessary because the drones still need low-latency, active human control.
Long range fixed wing kamikaze drones are usually autonomous, but even for those there were reports that taking remote control with FPV goggles can significantly increase accuracy and success rate.
When AI is developed that can control a drone well enough, and runs on a chip that's economical to put on kamikaze drones, it's going to be a game changer. [1]
Compared to that, I don't see how recharging en-route will change anything. For fiber-optic drones, landing and waiting for an ambush is already an established tactic. Sitting on a power line instead of the ground is going to make your drone much easier to notice.
For fixed wing drones, Russia's Shaheds can already easily cover Ukraine, they don't need more range.
1. Game changer in the "fucking horrifying" sense of course. ↩︎
4
My first thought was Amazon leveraging this for drone delivery.
1
I have raised this twice with UKR over the last year. Surprised they havn't done it yet.
-4
I like the name.
Sometimes in a computer program, it is important that separate portions be changed at the same time if they are ever changed. An example is batch size: if you have your batch size of 16 dotted throughout your program, changing batch size will be slow and error prone.
The canonical solution is “Single source of truth.” Simply store BATCH_SIZE=16 at the top of your program and have all other locations reference the value of this variable. This solves both the slowness and the error-prone-ness issues.
However, single source of truth has a complexity cost,...
Epistemic status: 11 pages in to “The lathe of heaven” and dismayed by Orr
Are alignment methods that rely on the core intelligence being pre-trained on webtext sufficient to prevent ASI catastrophe?
What are the odds that, 40 years after the first AGI, the smartest intelligence is pretrained on webtext?
What are the odds that the best possible way to build an intelligent reasoning core is to pretrain on webtext?
What are the odds that we can stay in a local maximum for 40 years of everyone striving to create the smartest thing they can?
My mental model o...
Vibe check: what about doing a "fire drill" where a random oracle is set up to fire at some point in a 1 week period, and when the oracle goes off everyone across different companies turns off their clustered gpus for ~5 minutes? Conditioning on the worlds where we get multiple chances at alignment, practicing this sort of coordination seems like it increases the number of chances we get.
What should I do if I had a sudden insight, that the common wisdom was right the whole time, if maybe for the wrong reasons? The truth- the honest to god real resolution to a timeless conundrum- is also something that people have been loon-posting to all comments sections of the internet. Posting the truth about this would be incredibly low status. I know that LessWrong is explicitly a place for posting low status truths, exactly as long as I am actually right, and reasoning correctly. Even though I fit those conditions I still fear that I'm going too far....
2
From your description, I have no idea what you mean by "treadmill speed dynamically matches the wheel radius * wheel angular velocity". From your conclusion, I can guarantee that it doesn't mean anything that matches most other people's constraints. Did someone somewhere post a particularly bad physical model that you're drawing on?
I’m working on a theory post about the conjunction fallacy, and need some manifold users to bet on a pair of markets to make a demonstration more valid. I’ve put down 150 mana subsidy and 15 mana of boosts, anyone interested?
https://manifold.markets/HastingsGreer/pa-pa-b-experiment-statement-y?r=SGFzdGluZ3NHcmVlcg
https://manifold.markets/HastingsGreer/pa-pa-b-experiment-statement-x?r=SGFzdGluZ3NHcmVlcg
Alignment plan: we nail a golden dubloon to the mast and promise it to the first researcher who spots a super intelligence, and then when we see it, we all get in little wooden boats and paddle after it with harpoons and ropes.
1
10
Ahah, I’ve got the proof here (Fedallah et al), with this alignment scheme its literally impossible for us to die, verified with Lean! (based on the axiom that we won’t see two hearses, one grown in america and the other not made by human hands. but like, what are the odds of that?)
-5
We've played "Pokemon or Tech Startup" for a couple years now. I think there's absolutely potential for a new game, "Fantasy Magic Advice" or "LLM Tips and Tricks." My execution is currently poor- I think the key difference that makes it easy to distinguish the two categories is tone, not content, and using a Djinn to tone match would Not Be In the Spirit of It. (I have freely randomized LLM vs Djinn)
Absolutely do not ask it for pictures of kids you never had!
My son is currently calling chatgpt his friend. His friend is confirming everything and has ...
2
Obviously the incident when openAI’s voice mode started answering users in their own voices needs to be included- don’t know how I forgot it. That was the point where I explicitly took up the heuristic that if ancient folk wisdom says the Fae do X, the odds of LLMs doing X is not negligible.
I feel like people are under-updating on the negative space left by the Deepseek r1 release. Deepseek was trained using ~$6million marginal dollars, Liang Wenfeng has a net worth in the billions of dollars. From whence the gap?
Lets examine an entirely prosaic situation: Carl, a relatively popular teenager at the local highschool, is deciding whether to invite Bob to this weekend's party.
some assumptions:
- While pondering this decision for an afternoon, Carls's 10^11 neurons fire 10^2 times per second, for 10^5 seconds, each taking in to account 10^4 input synapses, for 10^22 calculations (extremely roughly)
- If there was some route to perform this calculation more efficiently, someone probably would, and would be more popular
The important part of choosing a party invite as the task ...
5
I think your calculations are off by orders of magnitude. Not all neurons fire constantly at 100 times per second - https://aiimpacts.org/rate-of-neuron-firing/ estimates 0.29 to 1.82 times per second. Most importantly perhaps, not all of the processing is directed to that decision. During those hours, many MANY other things are happening.
1
Thanks for the link to the aiimpacts page! I definitely got the firing rate wrong by about a factor of 50, but I appear to have made other mistakes in the other direction, because I ended up at a number that roughly agrees with aiimpacts- I guessed 10^17 operations per second, and they guess .9 - 33 x 10^16, with low confidence. https://aiimpacts.org/brain-performance-in-flops/
3[anonymous]
and would be more popular
Not necessarily. In high school politics, pure looks, physical form, and financial support from the parents, all of which are essentially unrelated to brain processing, account for a significant chunk.
Popular media reference: look at Jersey shore, which is essentially the high school politics turned up. Many of the actors used very simple strategies, such as Snooki wandering around drunk and saying funny things, or Ronnie essentially just doing plenty of steroids and getting into endless fights.
Other than making sure the robotics hardware looks good, an AI algorithm could be dramatically more compact than the example you gave by developing a "popularity maximizing" policy from the knowledge of many other robots in many other high schools. Most likely, Carl is using a deeply suboptimal policy, not having seen enough training examples in his maximum of 4 years of episodes. (unless he got held back a year). A close to optimal policy, even one with a small compute budget, should greatly outperform Carl.