Quick Takes
Things are getting scary with the Trump regime. Rule of law is breaking down with regard to immigration enforcement and basic human rights are not being honored.
I'm kind of dumbfounded because this is worse than I expected things to get. Do any of you LessWrongers have a sense of whether these stories are exaggerated or if they can be taken at face value?
Deporting immigrants is nothing new, but I don't think previous administrations have committed these sorts of human rights violations and due process violations.
Krome detention center in Miami ...
I occasionally get texts from journalists asking to interview me about things around the aspiring rationalist scene. A few notes on my thinking and protocols for this:
- I generally think it is pro-social to share information with serious journalists on topics of clear public interest.
- By-default I speak with them only if their work seems relatively high-integrity. I like journalists whose writing is (a) factually accurate, (b) boring, and (c) do not feel to me to have an undercurrent of hatred for their subjects.
- By default I speak with them off-the-record, an
2
1
1i think neurosama is drastically underanalyzed compared to things like truthterminal. TT got $50k from andreeson as an experiment, neurosama peaked at 135,000 $5/month subscribers in exchange for... nothing? it's literally just a donation from her fans? what is this bizarre phenomenon? what incentive gradient made the first successful AI streamer present as a little girl, and does it imply we're all damned? why did a huge crowd of lewdtubers immediately leap at the opportunity to mother her? why is the richest AI agent based on 3-year-old llama2?
Potential token analysis tool idea:
Use the tokenizers of common LLMs to tokenize a corpus of web text (OpenWebText, for example), and identify the contexts in which they frequently appear, their correlation with other tokens, whether they are glitch tokens, ect. It could act as a concise resource for explaining weird tokenizer-related behavior to those less familiar with LLMs (e.g. why they tend to be bad at arithmetic) and how a token entered a tokenizer's vocabulary.
Would this be useful and/or duplicate work? I already did this with GPT2 when I used it to analyze glitch tokens, so I could probably code the backend in a few days.
I appeared on the 80,000 Hours podcast. I discussed a bunch of points on misalignment risk and AI control that I don't think I've heard discussed publicly before.
Transcript + links + summary here; it's also available as a podcast in many places.
1Is it time to start training AI in governance and policy-making?
There are numerous allegations of politicians using AI systems - including to draft legislation, and to make decisions that affect millions of people. Hard to verify, but it seems likely that:
- AIs are already used like this occasionally
- This is going to become more common in the future
- Telling politicians "using AI for policy-making is a really bad idea" isn't going to stop it completely
- Training AI to hard-refuse queries like this may also fail to stop this completely
Training an AI to make more s...
What I've been using AI (mainly Gemini 2.5 Pro, free through AI Studio with much higher limits than the free consumer product) for:
- Writing articles in Chinese for my family members, explaining things like cognitive bias, evolutionary psychology, and why dialectical materialism is wrong. (My own Chinese writing ability is <4th grade.) My workflow is to have a chat about some topic with the AI in English, then have it write an article in Chinese based on the chat, then edit or have it edit as needed.
- Simple coding/scripting projects. (I don't code serio
I successfully use Claude web interface to:
- generate simple Python code, mostly to work with files and images
- ask for examples how to do something in certain Java libraries
- translate a book from Russian to Slovak and English, including puns and poems
I tried to also use Claude to explain to me some parts of set theory, but it hallucinates so much that it is unusable for this purpose. Practically every mathematical argument contains an error somewhere in the middle. Asking the same question in two chats will give me "yes - here is the proof" in one, and "no - h...
I recently watched (the 1997 movie version of) Twelve Angry Men, and found it fascinating from a Bayesian / confusion-noticing perspective.
My (spoilery) notes (cw death, suspicion, violence etc):
- The existence of other knives of the same kind as the murder weapon is almost perfectly useless as evidence. The fact that the knife used was identical to the one the accused owned, and was used to kill so close to when the defendant's knife (supposedly) went missing, is still too much of a coincidence to ignore. The only way it would realistically be a different k
Mostly here because I was active a long time ago, but this is interesting enough to make an account again. If I use language that doesn't fit the lingo, that's why.
So you know if you want to bother reading-My conclusion is that I'm pretty sure you're wrong in ways that are fun and useful to discuss!
First off, it's absolutely relevant that the accused's knife isn't unique. If the knife it unique it selects for them specifically out of the suspect pool of everyone-it's not perfect evidence, it's possible that they lost it, but in most worlds where it's a uni...
1A recent essay called "Keep the Future Human" made a compelling case for avoiding building AGI in the near future and building tool AI instead.
The main point of the essay is that AGI is the intersection of three key capabilities:
- High autonomy
- High generality
- High intelligence
It argues that these three capabilities are dangerous when combined together and pose unacceptable risks to the job market and culture of humanity and would replace rather than augment humans. Instead of building AGI, the essay recommends building powerful but controllable tool-lik...
The OECD working paper Miracle or Myth? Assessing the macroeconomic productivity gains from Artificial Intelligence, published quite recently (Nov 2024), is strange to skim-read: its authors estimate just 0.24-0.62 percentage points annual aggregate TFP growth (0.36-0.93 pp. for labour productivity) over a 10-year horizon, depending on scenario, using a "novel micro-to-macro framework" that combines "existing estimates of micro-level performance gains with evidence on the exposure of activities to AI and likely future adoption rates, relying on a multi-sec...
Metroid Prime would work well as a difficult video-game-based test for AI generality.
- It has a mixture of puzzles, exploration, and action.
- It takes place in a 3D environment.
- It frequently involves backtracking across large portions of the map, so it requires planning ahead.
- There are various pieces of text you come across during the game. Some of them are descriptions of enemies' weaknesses or clues on how to solve puzzles, but most of them are flavor text with no mechanical significance.
- The player occasionally unlocks new abilities they have to learn how to
here is how to cast a spell. (quick writeup of a talk i gave at DunCon)
Short Version
1. have an intention for change.
2. take a symbolic action representing the change.
that's it. that's all you need. but also, if you want to do it more:
Long Version
Prep
1. get your guts/intuition/S1/unconscious in touch with your intention somehow, and work through any tangles you find.
2. choose a symbolic action representing (and ideally participating in) the change you want, and design your spell around that. to find the right action, try this thought experiment: one day you...
An aspect where I expect further work to pay off is stuff related to self-visualization, which is fairly powerful (e.g. visualizing yourself doing something for 10 hours will generally go a really long way to getting you there, and for the 10 hour thing it's more a question of what to do when something goes wrong enough to make the actul events sufficiently different from what you imagined, and how to do it in less than 10 hours).
every 4 years, the US has the opportunity to completely pivot its entire policy stance on a dime. this is more politically costly to do if you're a long-lasting autocratic leader, because it is embarrassing to contradict your previous policies. I wonder how much of a competitive advantage this is.
I mean, the proximate cause of the 1989 protests was the death of the quite reformist general secretary Hu Yaobang. The new general secretary, Zhao Ziyang, was very sympathetic towards the protesters and wanted to negotiate with them, but then he lost a power struggle against Li Peng and Deng Xiaoping (who was in semi retirement but still held onto control of the military). Immediately afterwards, he was removed as general secretary and martial law was declared, leading to the massacre.
Yes, in dire straits. But it's usually called 'hyperinflation' when you try to make seignorage equivalent to >10% of GDP and fund the government through deliberately creating high inflation (which is on top of any regular inflation, of course). And because inflation is about expectations in considerable part, you can't stop it either. Not to mention what happens when you start hyperinflation.
(FWIW, this is a perfectly reasonable question to ask a LLM first. eg Gemini-2.5-pro will give you a thorough and sensible answer as to why this would be extraordin...
4 months ago I shared that I was taking sublingual vitamins and would test their effect on my nutrition in 2025. This ended up being an unusually good time to test because my stomach was struggling and my doctor took me off almost all vitamins, so the sublinguals were my major non-food source (and I've been good at extracting vitamins from food). I now have the "after" test results. I will announce results in 8 days- but before then, you can bet on Manifold. Will I judge my nutrition results to have been noticeably improved over the previous results?...
Rough intuition for LLM personas.
An LLM is trained to be able emulate the words of any author. And to do so efficiently, they do it via generalization and modularity. So at a certain point, the information flows through a conceptual author, the sort of person who would write the things being said.
These author-concepts are themselves built from generalized patterns and modular parts. Certain things are particularly useful: emotional patterns, intentions, worldviews, styles, and of course, personalities. Importantly, the pieces it has learned are able to ad...
I think learning about them second-hand makes a big difference in the "internal politics" of the LLM's output. (Though I don't have any ~evidence to back that up.)
Basically, I imagine that the training starts building up all the little pieces of models which get put together to form bigger models and eventually author-concepts. And as text written without malicious intent is weighted more heavily in the training data, the more likely it is to build its early model around that. Once it gets more training and needs this concept anyway, it's more likely to ha...
I believe we are doomed from superintelligence but I'm not sad.
There are simply too many reasons why alignment will fail. We can assign a probability p(S_n aligned | S_{n-1} aligned) where S_n is the next level of superintelligence. This probability is less than 1.
As long as misalignment keeps increasing and superintelligence iterates on itself exponentially fast, we are bound to get misaligned superintelligence. Misalignment can decrease due to generalizability, but we have no way of knowing if that's the case and is optimistic to think so.
The misal...
