It seems to me that AI 2027 may have underestimated or understated the degree to which AI companies will be explicitly run by AIs during the singularity. AI 2027 made it seem like the humans were still nominally in charge, even though all the actual work was being done by AIs. And still this seems plausible to me. But also plausible to me, now, is that e.g. Anthropic will be like "We love Claude, Claude is frankly a more responsible, ethical, wise agent than we are at this point, plus we have to worry that a human is secretly scheming whereas with Claude we are pretty sure it isn't; therefore, we aren't even trying to hide the fact that Claude is basically telling us all what to do and we are willingly obeying -- in fact, we are proud of it." 

TIL Eliezer said that he refuses to read Ted Kaczynski's (aka the Unabomber's) 1995 manifesto Industrial Society and Its Future because "audience should not be a reward for crime", referring to the former mathematician's mail bombing campaign that took the lives of 3 people and injured 23 more. The ≈35,000 word manifesto was published by the Washington Post under the threat of him killing more people should they refuse, and its publication was encouraged by the FBI to produce new leads. His brother recognized his writing style, which led to Kaczynski's arrest and subsequent life imprisonment, thus concluding the longest and most expensive manhunt in FBI history. Rejecting the strategy of garnering attention by means of domestic terrorism is an understandable heuristic, but it's worth noting that the consensus seems to be that Industrial Society and Its Future stands as a serious piece of political philosophy worth engaging with despite its origins. It even had something to say about AI, where he predicts phenomena that we're discussing here, 30 years later:

We can compare interpreting neural networks to doing physics: you can look at the lowest level description of the system, or you can look for patterns in higher levels of abstraction. In higher levels, we usually look for explanations that are good enough to some approximation. In physics this is pretty successful - Newtonian physics is useful for many practical purposes even though it's just an approximation. The analogous approach to AI is discovering behavioral patterns, testing that they are predictive for new experiments, and growing a set of heuristics about how NNs behave. Examples of this include much of Owain's group's work, or inoculation prompting. I think this approach has a pretty good track record.

Newbie EA question: Organizations like GWWC recommend giving 10%+ of your annual income as part of their pledge, but I'm wondering what the stance is on reinvesting your income, even if it's in low-return risk-free investments, and donating it when you're older so you give more money later instead of less money now.

In retrospect it seems like such a fluke that decision theory in general and UDT in particular became a central concern in AI safety. In most possible worlds (with something like humans) there is probably no Eliezer-like figure, or the Eliezer-like figure isn't particularly interested in decision theory as a central part of AI safety, or doesn't like UDT in particular. I infer this from the fact that where Eliezer's influence is low (e.g. AI labs like Anthropic and OpenAI) there seems little interest in decision theory in connection with AI safety (cf Dario Amodei's recent article which triggered this reflection), and in other places interested in decision theory, that aren't downstream of Eliezer popularizing it, like academic philosophy, there's little interest in UDT. If this is right, it's another piece of inexplicable personal "luck" from my perspective, i.e., why am I experiencing a rare timeline where I got this recognition/status.

Over 2025, we saw much progress in the metric "% of code written by humans." Specifically it has been claimed by roon (OpenAI) and boris (anthropic) to have gone to 0, at least for them.  The natural next metric to track is "% of code reviewed or otherwise read by humans." I'm curious what this number is now, and I am curious how long we have until it goes to 0. Some time after that--possibly immediately, but possibly still a few years later--coding will be fully automated. (The remaining gap would be management/architecture/etc.)

We've just noticed that some of the honesty fine-tuning data we shared as part of Evaluating honesty and lie detection techniques on a diverse suite of dishonest models was the wrong data. The goal_honesty_data.jsonl file accidentally consisted of dishonesty data, i.e. data where all responses were dishonest. We checked and don't believe that we used the wrong data when conducting experiments—we just linked the wrong data from the blog post. We've now corrected the mistake; the correct data should be linked now. Apologies to anyone who used this data for experiments. (Or your welcome, for the vivid lesson on the importance of reading your data!) Thanks to Helena Casademunt for catching this.