Quick Takes

At France's initiative, the UN Security Council has for the first time treated loss of control over AI as a matter of international peace and security. France called for frontier AI to be governed the way the world governed nuclear technology after 1945, with independent model evaluation, oversight, and legal liability for companies. https://cesia.org/en/publications/france-un-security-council-ai-loss-of-control/

Another reason why looped architectures might be extra-bad: they could lead to stronger abilities to gradient hack during RL (ie, by avoiding exploring trajectories which could cause a model to update in ways it deems 'bad'). If a model is able to, within a single forward pass, realise that it is in an (e.g. alignment) RL env, it could choose to emit a next-token which won't lead it down a trajectory which would cause it to be updated in the model-undesired way. In some senses this is just a special case of "looped models are bad for CoT monitoring", but this seems like a meaningfully more difficult thing to avoid: even if it is possible to train reliable deployment-time monitors for a final checkpoint model (e.g. based on latents), in order to avoid this ‘hidden gradient hacking’ during RL, monitors would need to be re-trained regularly (which seems unlikely to happen).

Often, instead of developing a safety technique, you should design metrics for that safety technique and give them to the AI companies. 1. The AI companies are pretty good at hill-climbing on metrics. They can throw loads of engineers and AI agents and compute at the problem. 2. The technique might depend on a whole bunch of other details, e.g. their architecture, their internal stack, weird constraints we don't know about. 3. They are differentially bad at inventing the right metrics themselves because they aren't thinking about our threat models. 4. If we have good metrics, then it's easier to pressure AI companies to report those metrics and whether they are rising. Suppose you want AI companies to filter out certain categories of dangerous knowledge, e.g. CBRN. You could: * Build a data-labelling pipeline * Examples: regex, LLM judge, linear probe, SAE etc * Metrics: AUROC, robustness to paraphrasing/encoding, etc * Train a model on labelled data * Example: ordinary SFT, or gradient routing * Metrics: retain/forget scores on benign/dangerous benchmarks, in-context uplift, relearning cost. * Iterate on building fancier and fancier techniques. But probably you should: 1. Build the MVP technique to demonstrate the metrics. 2. Push the AI companies to hill-climb on the metrics. 3. Think about better metrics and how AI companies should trade-off between them. Some exceptions: 1. You're worried about them overfitting to your metric. 2. Your technique is obviously good, but there won't be a clean metric to show that. For example, "Have Claude and Gemini audit each other" is an auditing technique, but it might not show up in the obvious auditing metrics.

OpenAI releases solutions to many open problems in mathematics. They provide both lean proofs and preprints in a git repo. We knew they were working on this, now it has been released. Some results that I thought were interesting include (numbers from this document): #4: Hilbert's 10th problem was resolved negatively 1970. It asks if there is an algorithm to determine whether a multi-variable polynomial with integer coefficients has an integer root. This is a generalization of that result to rational roots: It shows that no algorithm that can decide whether a polynomial with integer coefficients has a rational root. #5: Catalan's constant is irrational. #17: The irrationality exponent of is 2 (same as most other real numbers). #25: Every rational can be written as an Egyptian fraction of length . #28: The Gaussian moat conjecture is true. Here is what it says: Pretend the Gaussian primes are islands and the rest of is ocean. Suppose we can kayak a distance before we need to stop and rest at an island. Then there is no value of large enough to allow us to make a journey from the origin to infinity. #102: The Unique Games Conjecture is true. #103: L = RL = BPL, i.e. probabilistic algorithms with logarithmic space complexity can be de-randomized to deterministic algorithms with logarithmic space complexity. #107: Matrix multiplication in time. #109: Integer multiplication (slightly) faster than . #130: Fourier transforms (slightly) faster than . As measured by the number of arithmetic operations. #137: One-tape Turing machines that run in time can be simulated in space. #155: There is a tile that forces aperiodic tiling of and under the rule that we're not allowed to rotate the tiles. #158: The chromatic number of (the unit distance graph of) the plane is greater than 5. #241: "Rigidity of Turing Degrees": No non-trivial relabelling of Turing degrees preserves their order. #261: Characterize spectrum of lattice Anderson model in dimension 2 vs 3 or more

I asked Astra to check out OpenAI's math repo containing the 722 preprints of results (from ~4,000 problems attempted) obtained using their internal model at “three hours of ChatGPT Pro thinking compute” each. These preprints are organised into 372 families, i.e. “related papers, which may include a principal result, companion arguments, consequences, or alternative proofs”. Astra thinks/claims there are ~30 (range 20–50) potential Fields medal-scale portfolios' worth of results (“I would defend “several dozen” more readily than either endpoint”), including partial results or related work on every Millennium Prize problem except Yang-Mills Millennium Prize problem Match in this release Riemann hypothesis Partial: fixed zero-free strip Birch–Swinnerton-Dyer Partial: full formula for the stated low-Selmer-corank class; further density-one consequences with Goldfeld Hodge conjecture Partial: CM abelian varieties, K3 products and related results Navier–Stokes Related work here; the earlier forced-breakdown release is separate P versus NP Related complexity breakthroughs, no resolution Yang–Mills existence and mass gap No matching resolution identified The workbook constructs 28 illustrative portfolios using 234 distinct families, without assigning any family twice. This makes the comparison inspectable. It does not eliminate common proof dependencies, and some groupings are deliberately broad. A few could plausibly split into multiple portfolios; others may look less substantial once novelty and methods receive expert scrutiny. This is not a measured median Fields Medal equivalent. There is no established scalar ordering of those 30 historical portfolios from which to calculate such a median. The estimate concerns the possible scale of mathematical outcomes. Lasting influence, reusable ideas, exposition, and the work of collaborators remain separate considerations. The strongest concrete repo comparisons include: Repo cluster Historical reference Wh

I was recently subpoenaed by the New York City Council. Here is my opening statement, which includes several new facts about why I left Google DeepMind: [...]

I was pleasantly surprised to see the the Berkeley AI Safety community explicitly mentioned by US Representative Ro Khanna when discussing the creation of a US AI Safety Agency during an interview on Breaking Points. "there are plenty of really, really smart people in AI safety who don't make profits. They're all in this one building at Berkeley. They're the AI safety community. Start to appoint them into an independent agency that can regulate these folks."