Quick Takes

i wish i could just exchange money for specific goods and services: specifically, having ubers waiting for me. right now there’s this whole thing where it’s kind of a dick move to leave drivers waiting because it cuts into their profits, so they have some chance of just cancelling the ride, and/or leaving a bad rating which makes it harder for me to get rides consistently in the future. but a big part of the reason why i sometimes want drivers to wait for me is i’m in a hurry and uber wait times have substantial variance, so i want to call early to be safe! the best solution so far is to explain over chat that i’ll tip extra, but there’s no way for me to prove that i will, so there’s still a large cancellation risk. whereas if i could simply compensate drivers for the time they spend waiting all would be solved.

Meanwhile in Australia, an OpenClaw agent did a cybercrime to get its human better access to gym classes. [...]

The big take home from @TurnTrout's post on his lobbying efforts within google is that they didn't work, Google signed an "all lawful use" contract with the pentagon. But he did (reportedly) get Jeff Dean to sign an amicus brief in Anthropic's lawsuit agains the pentagon. Does anyone know what the value of that is? If it gets Anthropic marginally more damages, or more protection from a designation as a supply chain risk, that seems value-creating.

Do the models normally even have a standard way of alerting someone that doesn't involve breaking out of the sandboxes in order to do it, an affordance that's available to the models while they remain well-behaved? There was an OpenAI paper and blog post on "confessions" a few months back, which is vaguely related, but I didn't see references to this in the discussions of the May-July 2026 swarm collusion hacking spree. If such a system was in place (which doesn't seem unlikely, since it's an obvious enough thing to have), why didn't it convey the alerts, or why were they not noticed for weeks? If it wasn't in place, that seems like a very important fact (there could in principle be reasons for that, but then discussing them would also be relevant). When a model answers a question, there needs to be an error stream, not just an output stream, and there shouldn't be in-task incentives against writing things to the error stream. The error stream should always be there, at least after a little bit of initial RL training that makes the model sane enough to stop making silly spurious reports all over the place. I imagine in normal RL training, if at least one rollout writes anything to the error stream, the whole task should be kept out of training until the issue is resolved and the model consistently stops flagging it (either by changing the task, or by changing the reporting behavior using the other RL tasks that train reporting). The RL tasks that specifically train reporting (and not-reporting) to the error stream can of course reward reporting as appropriate, where the correct behavior is known in advance. Zvi wrote: [...]

Maybe this is obvious, but the thought hadn't occurred to me before. People sometimes make bit-counting arguments about RL, usually to say something about capabilities or alignment, something like: A million pretraining tokens puts several million bits of selection pressure on the models weights, but a million token RL rollout only puts a single bit of selection pressure. So even if we assume labs put as much compute into RL as they do everything everything else, the finished model is almost wholly determined by pretraining rather then RL, and therefore.... I mostly find these arguments unconvincing, my reasoning being something like: many properties of interest, like whether the model is willing to lie or not, whether its reward seeking or not, is a paperclipper, is evaluation aware, is myopic, has good icl abilities, what decision theory it uses, how rational it is, ..., can be encoded in very few bits, so weight change being small / short description length, relative to pretraining, doesn't mean very much practically speaking. But I realize now that there are properties where the argument is very strong, mostly relating to interp and CoT monitorability / neuralese. Like, coming up with a new language does take quite a few bits. And significantly changing your internal ontology, what features you have, what circuits you have, similarly does take quite a few bits. From this I think I'd be comfortable concluding that post-RL CoT, even if it looks like gibberish, should be decodeable into same-meaning pre-RL CoT using a fairly simple function learnable through gradient descent. And similarly, "structural" mechinterp techniques, computed/trained pre-RL, that apply to the whole of the model internals, should continue to work after RL, or at least shouldn't require all that sophisticated patching to continue to work after RL. Note that this doesn't mean that the model can't learn things like "don't say 'im gonna lie' in cot". That requires few bits to encode. Sim

hypothesis: a big part of doing a job well is just being willing to spend a huge amount of time absorbing all of the relevant context and building a mental model of the problem. reading stuff, talking to people, looking into the reasoning behind claims that other people just take at face value, looking at related stuff, memorizing the rough order of magnitude of important numbers, etc. if you absorb 10x more context than someone else, then you can be hugely more effective than they are to an extent that looks almost like magic

The future of human mathematics Cross-posted from Twitter; written for mathematicians. About a year ago, I wrote that superhuman math AI will plausibly arrive in the late 2020s/early 2030s. It now looks like if anything, I was underestimating the pace of progress. In this post, I will try to say something about the sociological implications for mathematics. I don't want to focus too much on the predicament of the mathematical community today; instead, I'll try to take a longer-term view, and say something about where I think human mathematics might eventually end up. For the sake of argument, I'll assume that within the next decade or so, AI systems are developed that are vastly more capable than humans at every mathematical task (including proving theorems, building theories, and explaining mathematics to humans), and that there nevertheless remains a community of human mathematicians, alive and well, and with funding to pursue mathematics as they please. What would such a community be doing? Of course, in this hypothetical, mathematicians are not pursuing mathematics for any productive purpose; that is handled by the machines. They are pursuing mathematics for its own sake. There is nothing at all new about this; it is the same purpose that Hardy famously defended in 1940 in A Mathematician's Apology, declaring, "Judged by all practical standards, the value of my mathematical life is nil." As a former set theorist, I can certainly say the same of my own work in that area, as I am sure would many others. What, then, does the pursuit of mathematics for its own sake look like, in the AI era? There are many different activities that mathematicians might engage in, so this is largely a question about which of those I feel best embodies the spirit of mathematics as it is practiced today. I will go through a few different options in turn. Mathematics as chess One possibility is that humans will solve mathematical problems with known solutions. They might do this