Quick Takes

My model, for years now, has been that AI Safety at frontier labs is essentially a bunch of shallow instincts/behaviors covering up an ultimately Pythian power-maximizing entity. Now seems like a good time to outline that model. ---------------------------------------- Concrete framing First, by "AI Safety" I mean all of: * Employees focused on working on AI alignment and control. * Company-wide mindsets/narratives regarding being safety-conscious and responsible. * Specific people's self-narratives/self-images about what they are doing, especially the leadership's. * Internal projects focused on AI Safety. * Internal policies that prioritize safe and responsible R&D practices. * External policies regarding how to relate to regulations, the public, and other frontier labs. * Et cetera; the entire cluster of abstract objects relating to AI Safety. Those AI Safety objects do influence what the lab does, to some extent. But my current model is that, if any of them come into meaningful conflict with what would maximize the lab's power/ability to advance capabilities, they would be bent or discarded every time. As in: * The lab would have an internal narrative of being safety-conscious and benevolent. The lab would often take actions in line with this narrative, just by inertia. But that self-narrative will always twist to explain any action the lab takes as being justified, often via galaxy-brained reasoning, even if the action originates from hollow power-maximization and blatantly contradicts being responsible. * The lab will instinctively engage in various AI Safety research projects, as long as those projects are cheap enough that funding them won't meaningfully cut into the funding for capability scaling.[1] But if a given safety project, no matter how promising, will cost a lot, resources would not be directed to it. * See: OpenAI's Superalignment being denied the promised compute. * The lab would employ tons of people genuinely thinking of the

In the future, there will be millions, and then billions, and then trillions of broadly superhuman AIs thinking and acting at 100x human speed (or faster). If all goes well, what might it feel like to live in the world as it undergoes this transformation? Analogy: Imagine being a typical person living in England from 1520 to 2020 (500 years) but experiencing time 100x slower than everyone else, so to you it feels like only five years have passed: Year 1 (1520–1620). A year of political turmoil. In February, Henry VIII breaks with Rome. By March, the monasteries are dissolved. In May, Mary burns Protestants; by the end of May, Elizabeth reverses everything again. Three religions of state in the span of a season. In September, the Spanish Armada sails and fails. Jamestown is founded around November. The East India Company is chartered. But the texture of life is identical in December to what it was in January. You still read by candlelight, travel by horse, communicate by letter. Your religious opinions may have flip-flopped a bit but you are still Christian. The New World is interesting news but nothing more. Year 2 (1620–1720). In March, civil war breaks out. By April, the king is beheaded — a man who ruled by divine right, executed by his own Parliament! In June, the Great Plague sweeps London, killing a quarter of its population. Weeks later, the Great Fire burns it to the ground. In September, Newton publishes the Principia, recasting the universe as a mechanism of mathematical laws. The Glorious Revolution replaces one king with another, this time by Parliament's invitation, with a Bill of Rights attached. In the moment, the political event feels bigger. Later you’ll realize Newton mattered more. Newcomen builds a steam engine in November. It pumps water out of mines. You don't see what the hype is about. Year 3 (1720–1820). The last year in which you will feel at home in the world. In May, the Seven Years' War makes Britain the dominant global power; the Ne

Nvidia Rubin FP8 performance is 3.5x that of Blackwell, compared to only 1.6x for BF16. For Hopper and Blackwell, FP8 performance used to be 2x that of BF16 performance (for the same chip), but for Rubin it's 4.4x. Nvidia is betting on FP8 at a cost for BF16, which suggests the largest models can now be reliably pretrained in FP8. Hardware that abandons full support of BF16 performance will be significantly more performant in FP8. TPUv7 (Ironwood) still maintains a 2x performance ratio between FP8 and BF16, but it's Blackwell's contemporary, so it's interesting what's going to happen with TPUv8 (which will be Rubin's contemporary). In terms of (dense) FP8 performance per GW of all-in IT power, Rubin NVL72 [1] gives about 5.6e21 FLOP/s per GW, GB300 NVL72 about 2e21 FLOP/s per GW, TPUv7 about 3.5e21 FLOP/s per GW. Blackwell is a 4nm chip, while both Rubin and TPUv7 are 3nm chips, which might explain how TPUv7 FP8 performance per GW is in the middle, better than Blackwell, even though like Blackwell it doesn't abandon BF16, but worse than Rubin, since Rubin does take advantage of partially abandoning BF16. (SemiAnalysis estimates 225 kW per rack for Rubin NVL72 (3.1 kW per chip) in total IT power, and 180 kW per rack for GB300 NVL72 (2.5 kW per chip). For $42bn of the Anthropic contract, SemiAnalysis estimates that 600K TPUv7 chips rented through GCP will need 788 MW of IT power, 1.31 kW per chip. FP8 performance is 5e15 FLOP/s per chip for Blackwell, 17.5e15 FLOP/s for Rubin, 4.6e15 FLOP/s for TPUv7. The difference in power per chip between Rubin and TPUv7, both being 3nm chips, is striking and suspicious, suggesting a 2x compute die vs. package unit counting error, but the FLOP/s per GW numbers make sense, suggesting there is no error.) Hopper buildout happened in 2024, GB300 NVL72 is happening in 2026, Rubin NVL72 will be happening in 2027-2028, though it makes sense for the largest individual datacenter sites (more relevant for pretraining) to wait for Rubin U

It's looking like any AI that anyone in the US builds will be used for whatever purpose the government wants and modified to meet the government's needs. Anthropic refused to let the Department of War use their models to spy on Americans en masse or autonomously kill people, and for the past week the DoW has been trying to pressure them to change that. Today the Department of War said "If Anthropic doesn't let them use their models for any legal purpose the Pentagon wants ("all lawful use") the Pentagon will either cut ties and declare Anthropic a 'supply chain risk,' or invoke the Defense Production Act to force the company to tailor its model to the military's needs."[1] The DoW gave Dario until Friday to make a decision. The former would prevent any company that does business with the DoW from using Anthropic products,[2] something normally reserved for foreign adversaries, and has been threatened for a week. The latter is new, and IMO it's a big deal. It would mean no matter what Anthropic does, they can't control how the DoW uses their models. It's somewhat ambiguous whether this would be a legal use of the DPA; it's normally reserved for hardware not software. It's also not clear to me what it would meant to have Anthropic "tailor its model" to DPA usage. People at AI companies often assume if they trust the leadership of their company, then they don't have to worry about egregious misuse. But if they develop the technology, they can't stop the government form getting their hands on it. And this incident is evidence the government is very willing to take extreme measures to get their hands on it and that they intent to use it for things like spying on Americans en masse. I expected at some point when AI was very powerful governments would try to nationalize it, but I didn't expect this kind of action when the technology was this early along in development, when it was very far from posing a decisive strategic advantage. 1. ^ It's important to

RSP takes from a bunch of Astra fellows: * Seems like Anthropic should've known RSPv2 would fail when the RAND report came out, and in retrospect it's kind of embarrassing we (the community) didn't realize this earlier * We're very divided on whether the phrasing/stance on "Anthropic has to win" is good/correct, especially given the talk about "marginal risk" considerations. We're somewhat concerned that Anthropic simply won't pause when it's clear (to concerned parties internally) they probably should.  * Why don't they just say racing is bad and that a pause (at some point) would be good? This seems so low-cost to put in the intro/industry reccs., or at least to make an OOM more clear. * Are Anthropic employees not reacting to this? It feels surprisingly low-profile for such a big change in internal governance (although I suppose there are Other Things happening).  * Maybe Anthropic should've been more clear about what "behind" and "ahead" mean, and when or when not they're giving themselves the option/soft obligation to pause * In general, we're quite confused about Anthropic's viewpoints on the difficulty of alignment and the likelihood of AI takeover.  * Risk reports seem good! We are quite excited for these! But 6 months is way too long of an interval (3 months might be okay?), and we would be less nervous if there were many addendums + edits as models were deployed (and this seems to be the case!). Also, we are unconvinced this doesn't fail during software-only AI R&D takeoff. * On a personal note, many of us are much more nervous about working for Anthropic and are much more nervous about the strategic decision-making of its leadership during the critical period.  EDIT: OOM ==> order of magnitude (which isn't a lot because they didn't make it at all clear!)

Am I missing anyone/anything?

What conclusions do people draw from the Epstein files about the global elite? It seems like a kind of interesting and amazing window into a huge swath of wealthy and powerful people.  I haven't dug in that much yet, but so far I'm struck by how sleazy many of them seem and how rarely they push back against unethical behavior. And how bad at spelling and inarticulate people who, in other contexts, seem fairly intellectual (e.g. Larry Summers), are in their casual communications.  I'm unsurprised by the lack of evidence of other mass conspiracies.