This is a special post for quick takes by David James. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
David James's Shortform
117David James
16MondSemmel
2Eli Tyre
3David James
9FlorianH
2leogao
13David James
0StanislavKrym
3David James
3David James
1tryhard1000
3David James
2Viliam
1David James
1David James
1David James
1David James
1David James
1David James
2MichaelDickens
1David James
I'm noticing evidence that many of us may have an inaccurate view of the 1983 Soviet nuclear false alarm. I say this after reading "Did Stanislav Petrov save the world in 1983? It's complicated". The article is worth reading; it is a clear and detailed ~1100 words. I've included some excerpts here:
[...] I must say right away that there is absolutely no reason to doubt Petrov's account of the events. Also, there is no doubt that Stanislav Petrov did the right thing when he reported up the chain of command that in his assessment the alarm was false. That was a good call in stressful circumstances and Petrov fully deserves the praise for making it.
But did he literally avert a nuclear war and saved the world? In my view, that was not quite what happened. (I would note that as far as I know Petrov himself never claimed that he did.)
To begin with, one assumption that is absolutely critical for the "saved the world" version of the events is that the Soviet Union maintained the so-called launch-on-warning posture. This would mean that it was prepared to launch its missiles as soon as its early-warning system detects a missile attack. This is how US system works and, as it usually happens, most people automatically assume that everybody does the same. Or at least tries to. This assumption is, of course, wrong. The Soviet Union structured its strategic forces to absorb a nuclear attack and focused on assuring retaliation - the posture known as "deep second strike" ("ответный удар"). The idea was that some missiles (and submarines) will survive an attack and will be launched in retaliation once it is over.
[...] the Soviet Union would have waited for actual nuclear detonations on its soil. Nobody would have launched anything based on an alarm generated by the early-warning system, let alone by only one of its segments - the satellites.
[...] It is certain that the alarm would have been recognized as false at some stages. But even if it wasn't, the most radical thing the General Staff (with the involvement of the political leadership) would do was to issue a preliminary command. No missiles would be launched unless the system detected actual nuclear detonations on the Soviet territory.
Having said that, what Stanislav Petrov did was indeed commendable. The algorithm that generated the alarm got it wrong. The designers of the early-warning satellites took particular pride in the fact that the assessment is done by computers rather than by humans. So, it definitely took courage to make that call to the command center up the chain of command and insist that the alarm is false. We simply don't know what would have happened if he kept silence or confirmed the alarm as positive. And, importantly, he did not know it either. He just did all he could to prevent the worst from happening.
I haven't made a thorough assessment myself. For now, I'm adding The Dead Hand to my reading list.
Edited on 2025-12-25 to improve clarity of the first point. Thanks to readers for the feedback.
3
1Feedback: I had to read your first sentence like 3x until I understood you meant "based on reading this, I think many of us had an inaccurate view previously") instead of "I think many of us have an inaccurate view from reading this".
I also had this initial misreading, but I think I figured out what you meant on my first reread.
Sorry for the confusion. :P ... I do appreciate the feedback. Edited to say: "I'm noticing evidence that many of us may have an inaccurate view of the 1983 Soviet nuclear false alarm. I say this after reading..."
2I was always slightly suspicious of the claims that we had x (5 or so?) times been closer to entering the big nuclear war than to not entering it. But if this passage is accurate, then the fact that some of the usual claims are so easy to put into perspective would suggest in some communities we are also a bit more affected by sensationalism than I'd have thought. Interesting, thanks!
do we have strong reasons to believe the US early detection system was much less likely to have a similar false alarm? if not, then the Petrov incident should count somewhat as a near miss.
During yesterday's interview, Eliezer didn't give a great reply to Ezra Klein's question: i.e. "why does even a small amount of misalignment lead to human extinction." I think many people agree with this; still, my goal isn't to criticize EY. Instead, my goal is to find various levels of explanation that have been tested and tend to work for different audiences with various backgrounds. Suggestions?
Related:
speck1447 : ... Things get pretty bad about halfway through though, Ezra presents essentially an alignment-by-default case and Eliezer seems to have so much disdain for that idea that he's not willing to engage with it at all (I of course don't know what's in his brain. This is how it reads to me, and I suspect how it reads to normies.)
1I think that Elieser means that mildly misaligned AIs are also highly unlikely, not that a mildly misalinged AI would also kill everyone:
When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. So far as I'm concerned, if you can get a powerful AGI that carries out some pivotal superhuman engineering task, with a less than fifty percent change of killing more than one billion people, I'll take it.
As for LLMs being aligned by default, I don't have even the slightest idea on how Ezra even came up with this. GPT-4o has already been a super-sycophant[1] and driven people into psychosis in spite of OpenAI prohibiting it by their Spec. Grok's alignment was so fragile that xAI's mistake caused Grok to become MechaHitler.
- ^
In defense of 4o, it was raised on human feedback which is biased towards sycophancy and demands erotic sycophants (c) Zvi. But why would 4o drive people into a trance or psychosis?
Here is one easy way to improve everyday Bayesian reasoning: use natural frequencies instead of probabilities. Consider two ways of communicating a situation:
Probability format
- 1% of women have breast cancer
- If a woman has breast cancer, there's an 70% chance the mammogram is positive
- If a woman does not have breast cancer, there's a 10% chance the mammogram is positive.
Natural frequency format
- Out of 1,000 women, 10 have breast cancer
- Of those 10 who have cancer, 7 test positive
- Of the 990 without cancer, 99 test positive
For each of the two formats above, ask this question to a group of people: "A woman tests positive. What is the probability she has cancer?". Which do you think gives better results?
References
- Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: frequency formats. Psychological review, 102(4), 684.
- Hoffrage, U., Lindsey, S., Hertwig, R., & Gigerenzer, G. (2000). Communicating statistical information. Science, 290(5500), 2261-2262.
- Hoffrage, U., Gigerenzer, G., Krauss, S., & Martignon, L. (2002). Representation facilitates reasoning: What natural frequencies are and what they are not. Cognition, 84(3), 343-352.
Asking even a good friend to take the time to read The Sequences (aka Rationality A-Z) is a big ask. But how else does one absorb the background and culture necessary if one wants to engage deeply in rationalist writing? I think we need alternative ways to communicate the key concepts that vary across style and assumed background. If you know of useful resources, would you please post them as a comment? Thanks.
Some different lenses that could be helpful:
-
“I already studied critical thinking in college, why isn’t this enough?”
-
“I’m already a practicing data scientist, what else do I need to know and why?
-
“I’m already interested in prediction markets… how can I get better?”
-
“I’m not a fan of parables, can you teach me aspiring rationality in a different way?”
-
“I would never admit this except to close friends, but weird or fringe movements make me nervous, but as do think of myself as a critical thinker, help me ramp up without getting into uncomfortable areas.”
-
“I would never admit this to others — and may not even recognize it in myself — but my primary motivation is to use evidence and rationality as a cudgel against my opponents. Can you teach me how to be better at this?” (And this reminds me of Harry teaching Malfoy in HPMOR with an ulterior motive!)
As a STEM enthusiast, I suspect I would've much more quickly engaged with the Sequences had I first been recommended arbital.com as a gateway to it instead of "read the Sequences" directly.
Is power shifting away from software creators towards attention brokers? I think so...
Background: Innovation and Compositionality
How does innovation work? Economists, sociologists, and entrepreneurs sometimes say things like:
- new scientific ideas are born when the stars align
- the same idea pops up in many places because an idea is in the air
- some ideas become products by way of tech transfer
- pain points spur solutions
- software companies sometimes dogfood their own technologies to vet them
- hardware companies who understand and control the technology of their production pipeline often can better innovate at a process level
Software engineers would probably add to the list by saying practical innovation is driven by the awareness and availability of useful building blocks such as libraries, APIs, data structures, and algorithms. Software developers know this. These blocks are the raw material for their work.
But, I don't know if the rest of the world gets it. Off the top of my head, I don't think I've yet seen a compelling account of this -- how compositionally in software feels different completely bonkers compared with other industries. Just keeping up is exciting (if you are lucky) but often disorienting. If you look backwards, previous practices seem foreign, clunky, quaint, or even asinine.
Imagine if this rate of change applied to dentistry. Imagine a dentist sitting down with a patient. "Hello, how are you today?" The patient answers nervously, "Not too bad..." and mentally appends "...yet". The dentist says "Well, let's have a look-see..." and glances over at her tray of tools. Eeek. Nothing looks familiar. She nervously calls for an assistant. "Where is my Frobnicator???" The assistant answers: "That technology was superseded yesterday. Here. Use the Brofnimator instead."
Software development feels like this.
To the extent software is modular, components can be swapped or improved with relatively little cost. Elegant software abstractions reduce the cost of changing implementation details. It is striking how much intellectual energy goes into various software components. Given the size and scope of software industry, maybe we shouldn't be surprised.
Example: seemingly over the course of only a few years, there seems to a widespread acceptance (in circles I read, like Hacker News) that embedded databases can play key roles. Rather than reaching for, say, a server-based database management system (DBMS) like PostgreSQL, developers increasingly choose embedded (in-process) data storage libraries like SQLite or one of the many embedded K/V stores (RocksDB, LMDB, etc). Many of the newer K/V stores have been developed and adopted quite rapidly, such as redb.
Tension Between Building and Adoption
Now, to abstract a bit, here is my recent insight about software. When I think of the balance between:
-
the cost (time, money, labor) of designing & building, versus:
-
the steps needed to socialize, persuade, trial, adopt, integrate
... it is clear the cost (1) is dropping fast. And large parts of (2) are dropping too. The following are getting easier: (a) surveying options; (b) assessing fitness for purpose; (c) prototyping; (d) integrating.
This means that attention, socialization, persuasion are increasingly important. This kind of trend is perhaps nothing new in the domains of politics, advertising, fashion, and so on. But it seems notable for software technology. In a big way it shifts the primary locus of power away from the creators to the attention brokers and persuaders.
Yeah, the amount of change feels overwhelming to me, too. During my career as a software developer I have programmed in Basic, C++, Clojure, Java, JavaScript, Pascal, Perl, PHP, Python, Ruby, and XSLT. In Java alone, I wrote front ends in AWT, Java Server Faces, Java Server Pages, PrimeFaces, Struts, Stripes, Swing. As a database, I used Datomic, DynamoDB, H2, Microsoft SQL Server, MySQL, Oracle, PostgreSQL. Probably forgot some.
I don't mind learning new things, but it is annoying to spend a year or more learning something, only to throw it away and learn something different that accomplishes more or less the same thing, and then do it again, and again. Especially when there are so many genuinely useful things that I do not have the capacity to learn anymore.
Before I had kids, I learned new things in my free time; now I have less free time than I used to have, and I want to spend some of it doing things that are not related to my job.
...end of rant.
That said, I don't understand your question. Do you think that in future, AI will write the code, and the job of the developer will be to persuade the employer to use technology X instead of technology Y? Why not ask AI instead?
Is power shifting away from software creators towards attention brokers?
That said, I don't understand your question. Do you think that in future, AI will write the code, and the job of the developer will be to persuade the employer to use technology X instead of technology Y? Why not ask AI instead?
Upon further reflection, my question is more about function (or role) than who has the power. It seems like the function of persuasion is more important than ever, relative to creation. It might be helpful to think of the persuasion role being done by some combination of people, AI decision support, and AI agents.
The writing above could be clearer as to what I mean. Here are some different ways of asking the question. (I abbreviate software creator as creator.)
- Is the median creator less powerful today...
- overall?
- relative to the median influencer?
- Is the set of all software creators less powerful today...
- overall?
- relative to the set of all influencers?
Since there is variation across industries or products (call them value chains):
- For a particular value chain, is a median creator less powerful today relative to a media influencer?
Or maybe we want to ask the question from the POV of an individual:
- For a power-seeking individual, would they be better off becoming a creator or an influencer?
Piccione and Rubinstein (2007) have developed a ‘jungle model'. In contrast to standard general equilibrium models, the jungle model endows each agent with power. If someone has greater power, they can simply take stuff from those with less power. The resulting equilibrium has some nice properties, including that it exists, and that it is also Pareto efficient.
I’m about to start reading “Fifty Year of Research on Self-Replication" (1998) by Moshe Sipper. I have a hunch that the history and interconnections therein might be under-appreciated in the field of AI safety. I look forward to diving in.
A quick disclosure of some of my pre-existing biases: I also have a desire to arm myself against the overreaching claims and self-importance of Stephen Wolfram. A friend of mine was “taken in” by Wolfram’s debate with Yudkowsky… and it rather sickened me to see Wolfram exerting persuasive power. At the same time, certain of Wolfram’s rules are indeed interesting, so I want to acknowledge his contributions fairly.
I wanted to highlight the Trustworthy Systems Group at School of Computer Science and Engineering of UNSW Sydney and two of their projects, seL4 and LionsOS.
We research techniques for the design, implementation and verification of secure and performant real-world computer systems. / Our techniques provide the highest possible degree of assurance—the certainty of mathematical proof—while being cost-competitive with traditional low- to medium-assurance systems.
seL4 microkernel
seL4 is both the world's most highly assured and the world's fastest operating system kernel. Its uniqueness lies in the formal mathematical proof that it behaves exactly as specified, enforcing strong security boundaries for applications running on top of it while maintaining the high performance that deployed systems need.
seL4 is grounded in research breakthroughs across multiple science disciplines. These breakthroughs have been recognised by international acclaimed awards, from the MIT Technology Review Award, to the ACM Hall of Fame Award, the ACM Software Systems Award, the DARPA Game changer award, and more.
LionsOS
They are building a modular operating system called LionsOS:
We are designing a set of system services, each made up of simple building blocks that make best use of the underlying seL4 kernel and its features, while achieving superior performance. The building blocks are simple enough for automated verification tools (SMT solvers) to prove their implementation correctness. We are furthermore employing model checkers to verify key properties of the interaction protocols between components.
Core to this approach are simple, clean and lean designs that can be well optimised, use seL4 to the best effect, and provide templates for proper use and extension of functionality. Achieving this without sacrificing performance, while keeping the verification task simple, poses significant systems research challenges.
Here are two compound nouns that I've found useful for high-bandwidth communication: economics-as-toolkit versus economics-as-moral-foundation.
1To varying degrees, people immersed in the study of economics may adopt more than just an analytical toolkit; they may hold some economic concepts as moral foundations.
To give some context, I have bolded two parts from Teaching Economics as if Ethics Mattered (2004) by Charles K. Wilber:
I have spent the past thirty-five years as a professor of economics. Over the course of my tenure at The American University, in Washington, DC (1965-75) and the University of Notre Dame (1975 present), however, I became ever more disenchanted with the capacity of traditional economic theory to enable people to lead a good life. I found myself unable to accept the values embedded in economic theory, particularly the elevation of self-interest, the neglect of income distribution, and the attempts to export these values into studies of the family, the role of the state and so on. As a result I started researching and writing on the nature of economics and the role of ethics in economic theory.
This work has led me to three important conclusions. First is the conviction that economic theory is not value-free as is so often claimed. Rather, it presupposes a set of value judgments upon which economic analysis is erected. Second is the realization that the economy itself requires that the self interest of economic actors be morally constrained. Third is the recognition that economic institutions and policies impact people's lives requiring that both efficiency and equity be assessed. Teachers of economics need to make use of these insights.
Though Wilber (above) focuses on neoclassical economics as entangled with libertarianism, there are other couplings across different flavors of economic history: Marxism, Keynesianism, Development Economics, and more.
I have a working theory, which I have reflected on imperfectly: mentioning the difference between an analytical toolkit (aka modeling approach) and a value system goes a long way towards orienting myself in a discussion with someone I don't know well. In some ways, it can be a test balloon to assess someone's awareness of the complexity of economics.
Curated and popular this week