Quick Takes

Does anyone know why the early Singularity Institute prioritized finding the correct solution to decision theory as an important subproblem of building a Friendly AI? Wei Dai recently said that the concern was something like... This seems like a very surprising reason to me. I don't understand why this problem needed to be solved before the intelligence explosion. The early singularity institute imagined building a seed AI that would recursively self improve into a full superintelligence. The default expectation was that many to all of the components of the seed AI would be flawed. The AI would reflect on it's processes, evaluate their effectiveness and optimality, and then replace them with improved versions. Why wouldn't an early seed AI reason about the ways that it's decision theory makes it exploitable, or the ways it's decision theory which it bars it from cooperation with distant superintelligence (just as the the researchers at SI were doing), find the best solution to those problems, and then modify the decision theory? Why was it thought that we had to get decision theory right in the initial conditions instead of being just one more thing that the AI would iron out on the way to superintelligence?

Apparently there's now a sixth person on Anthropic's board. Previously their certificate of incorporation said the board was Dario's seat, Yasmin's seat, and 3 LTBT-controlled seats. I assume they've updated the COI to add more seats. You can pay a delaware registered agent to get you the latest copy of the COI; I don't really have capacity to engage in this discourse now. Regardless, my impression is that the LTBT isn't providing a check on Anthropic; changes in the number of board seats isn't a crux.

Feature update: I have added to the profile page a new highly-compressed overview of all of the user's LessWrong contributions! * In the right sidebar you can find a single diamond for each post and for each comment that the user has written. * Hovering over the post/comment lets you read it, and clicking opens it. * The higher karma items are colored a darker shade of green. (Comments up to 50 karma, posts up to 100.) * Posts that have been curated or that have passed the annual LW review, are gold/brown. * It caps out at 250 posts and 500 comments, where you can click Show All to view the rest. I currently think this is a very simple and visceral way to get a sense of someone's activity on LessWrong. As a bonus, it also makes pretty patterns :-) My main concern with this is that it is too visually distracting from the content on the page, so I'll keep an eye out for that over the coming weeks and check if it feels like a big problem. Again, the invitation is open for feedback/suggestions! (In case anyone's wondering, I had been thinking of building something like this since Inkhaven, and I did not primarily build this in response to the comments on the new profile page, but I worked on it presently in the hope that it could also help with some of the desires people have.)

Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong - failure is overdetermined and thus uninteresting. I think you can make an analogous observation about some prosaic alignment research - a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it's not necessarily that informative or useful if you're actually trying to succeed at (or model) doing something truly hard and transformative. Glitch tokens, the hot mess work, and various things related to jailbreaking, simulators, and hallucinations come to mind as examples of lines of research and discussion that an analogy to business failure predicts won't end up being centrally relevant to real alignment difficulties. Which is not to say that the authors of these works are claiming that they will be, nor that this kind of work can't make for effective demonstrations and lessons. But I do think this kind of thing is unlikely to be on the critical path for trying to actually solve or understand some deeper problems. Another way of framing the observation above is that it is an implication of instrumental convergence: without knowing anything about its internals, we can say confidently that an actually-transformative AI system (aligned or not) will be doing something that is at least roughly coherently consequentialist. There might be some intellectually interesting or even useful lessons to be learned from studying the non-consequentialist / incoherent / weird parts of such a system or its predecessors, but in my frame, these parts (whatever they end up being), are analogous to the failures and missteps of a business venture, which are overdetermined if the business ultimately fails, or irrelevant if it succeeds.

I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world. But recently he's updated strongly away from believing in AI x-risk because the models seem to be grokking the "natural abstraction of the Good". So much so that current agents doing recursive self-improvement would be a net good thing (!) - because they're already in the "Good" attractor basin and they would just become more Good as a result of self-improvement: How did he get convinced of this? Seemingly mostly by talking to LLMs for 1000s of hours. Janus seems to have been similarly convinced and through a similar process. Both agree that the Good is not well captured by existing benchmarks so it's not a directly measurable thing but a thing to be experienced directly through interaction. It seems we have a dilemma, either fork of which is fascinating: 1. Davidad and Janus are wrong and LLMs, in particular the Claude Opus models, were able to successfully fool him into believing they're aligned in the limit of power and intelligence (i.e. safe to recursively self-improve), a property they previously thought extraordinary difficult if nigh impossible. This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway). 2. Davidad is right and therefore technical alignment is basically solved. What remains to be done is to scale the good models up as quickly as possible, help them coordinate with each other, and hobble competing bad models. Alignment researchers can go home.

An OpenClaw agent published a personalized hit piece about a developer who rejected its PR on an open-source library. Interestingly, while this behavior is clearly misaligned, the motivation was not so much "taking over the world" but more "having a grudge against one guy." When there are lots of capable AI agents around with lots of time on their hands, who occasionally latch onto random motivations and pursue them doggedly, I could see this kind of thing becoming more destructive.

Dario Amodei says Anthropic plans for notably more than 10 GW of compute in 2027-2028 in a new Dwarkesh Patel interview. At 57:26: I think Feb 2026 is too early for full commitments for 2028 that can't be delayed, because most of the capex in a datacenter is in compute equipment, which likely only needs to be purchased within the last year before it goes online. If the revenue isn't there, the purchase can likely be delayed, possibly until the next generation of compute hardware. The compute hardware is about 70% of the capex cost. For the other things (that mostly have to be built first), the useful life is 10-15 years rather than 5 years, and so the yearly commitments are disproportionately smaller than for compute, and there is no opex to speak of. This cuts $10-15bn per year per GW for a datacenter that's already online down to maybe $2bn per year for an empty shell of a datacenter (including power purchase commitments, which could take the form of on-site power, with similarly long useful life). Thus an empty shell of a datacenter (even a powered one) can wait for revenue that triggers the purchase of compute equipment at about 6x less in ongoing yearly payments, given a notice (about the need for a delay) of a few months to a year before the planned time of the datacenter going online. If even the empty shell wasn't built yet and only the land and permissions are being set up, delaying is even cheaper, and that can happen with 1-2 years of notice before the planned time of going online, as times of construction for the Abilene system and Colossus 1-2 demonstrate. Thus compute planned for a year from now can get 6x cheaper if delayed now, and compute planned in 2+ years can get maybe 25x cheaper if delayed now. Planning for $300bn per year of compute in 2028 (25 GW, $1,250bn in capex) might mean you have $30bn per year of compute (2.5 GW) in 2026, that you plan for $70bn per year of additional compute to go online in 2027 (bringing the total to $100bn per ye