Quick Takes

1. It seems that Anthropic is now in the lead; they are IMO the most likely single company to automate AI R&D first. 2. I'm getting increasingly concerned about Anthropic's attitude towards alignment/safety. I grimly predict that they would basically behave like OpenBrain does in AI 2027, if they are lucky enough to get a pattern of misalignment evidence that egregious.

It's widely rumored that both OpenAI and Anthropic will go public within the next year or so, triggering massive capital inflows. I think this unleashes an underrated financial dynamics. An optimal portfolio hedges your personal risks. Unless you're independently wealthy, the risk of job loss due to AI automation is one your largest personal financial risks, and it makes a lot of sense to buy insurance against that (i.e., to buy into AI companies). Right now, individual investors can't do that. And the investors that have access to OpenAI/Anthropic on the private market are precisely the ones that don't need to hedge that risk. Once you open up to retail, a rational investor should park a sizable percentage of their net worth in those shares, opening up a capital gusher. This is especially true for Anthropic -- there's basically no way for a retail investor to get meaningful Anthropic exposure if they want it. It's a little different for OpenAI given that SoftBank is kinda an OpenAI proxy at this point (albeit in ways that make it a less attractive investment than the real thing). Seems like this could lead to substantial acceleration of progress. It may also change the politics. If stopping AI progress mostly means financially soaking billionaires, then it's politically a lot more palatable than if it will also screw up the 401ks of the mass affluent.

Trying to distill why strategy-stealing doesn't work even for consequentialists: Consider a game between A and B, where at most 1 player can win and: * U_A(A wins)=3, U_A(B wins)=2, U_A(both lose)=0 * U_B(A wins)=0, U_B(B wins)=3, U_B(both lose)=0 At time 1, A has a button that if pressed, ends the game and gives 40% chance of both players losing and 60% of A winning. A can press, pass, or surrender (giving B the win). At time 2, the button passes to B, who has the same options with "press" giving 60% chance of winning to B. At time 3 if both passed, they each have 50% chance of winning. Solving this backwards, at time 2, B should press because that gives U=.6x3 vs .5x3 for passing, so at time 1, A should surrender because U_A(press)=.6x3, U_A(pass)=U_A(B presses)=.6x2, U_A(surrender)=2. In terms of theory, this can be explained by this game violating the unit-sum (mathematically equivalent to zero-sum) assumption of strategy-stealing. It confuses me that it has significant mind-share among AI safety people, e.g. @ryan_greenblatt here, despite the world in general, and technological races in particular, obviously not being zero-sum. See also my failure to "steal" the strategy of investing in AI companies.

publicly registering a bet with Gabor Hollbeck: i predict that the median CS postdoc will be publishing less than 100 papers a year 5 years from now. Gabor predicts otherwise.

A way to think about Condensation in the algorithmic setting These are note that describe a (somewhat flawed) variation of algorithmic condensation. It covers similar ground as this post by Abram, with slightly more concrete detail. I'm publishing it because a) it's simple and relatively concrete and still works on most test cases I've worked through, and b) it helps motivate the more general approach. We want a generative model whose output is a set of observations. We want the hypothesis space to be the space of programs, and we want it to factorize naturally into components such that each component "contributes to" some subset of the observations. Observations: A set of tuples, where each tuple contains some content and a location. For example, the content might be a pixel r,g,b and the location might be the coordinates of that pixel x,y. Or the content might be a bit b and location as the index of that bit inside a large bitstring. To make the factorization work we need to impose some structure on the program. We'll have one top program which can access (via built-in instructions) a List of other strings. The List must have an interface that satisfies some properties: 1. It must be possible to retrieve elements based on their content, or their index in the list. 2. Item retrieval from the List is such that any items other than the one that actually ends up being retrieved can be removed from the List without changing the behavior of the program. For example, we could have a function lookup that scores each item on the list and returns the highest scoring one. The top program, when executed, must output the full set of observations (and no others). Scoring subsets A condensation has a score for every subset A of observation variables. Conditioned[1] score: Delete as many strings as possible from the List such that the subset of observations A is still recovered correctly by running the program. Sum the length of these programs. This gives the conditi

@jimrandomh recently argued me somewhat towards the position "it's time to start treating LLM Agents as moral patients." I'm not sure I'll actually represent his view, and I still feel pretty confused about it. Before I get started, I want to clarify: 1. I'm not actively claiming AIs are moral patients. Just, you should probably start treating them as such. 2. I don't think it's particularly obvious what "treat an AI like a moral patient" actually implies. They are weird aliens, their emitted words don't straightforwardly map to human preferences. But, basically, it seems like we're at the point where for any given property I would previously had said "you need to have this to be a moral patient", from the standpoint of watching inputs and outputs, LLM Agents seem to display each of those properties at least sometimes. From the outside, it seems to me that current AIs (or AIs + agent scaffolds), have a cluster of properties that seems drawn from the same urn that contains squids, pigs, dogs, infants and maybe toddlers, humans-with-some-kinds-of-mental-disabilities, and some flavors of imaginary sci-fi aliens. (In addition to Claude 4.6 just seeing all around MVP-AGI-ish, one thing that put me over-the-threshold for believing this are AI scaffolds, sometimes with multiple AIs working in concert where one has the sole job "check if the overall process is spiralling and reboot it if so." The way I think about each LLM-instance is more like "a process in a brain" than "the entire brain, necessarily.") People debate whether pigs or squids are moral patients, and whether portia-spiders and honeybees are more like "weakly-moral-patient-y" or "zero moral patient-y." It's not obvious. Also, people define morality all kinds of different ways, often while being fundamentally confused about it. But, I don't think you need to have it pinned down to start thinking "okay, maybe it's time to start thinking about these guys through a moral lens, especially if I'm interactin

Does anyone have a good theory of semiotic warfare? Like, when people start advocating for using words differently, when should you "cede" linguistic ground and when should you fight to keep the old definition? When if ever is it permissible to punish people who insist on using words in the new way, or the old way? Or if you don't like the normative frame - what is the realpolitik of semiotic warfare? How does one best achieve one's goals given the ongoing war over language? Obvious examples that come to mind are "alignment", "science", "God", "war crime", "woman", "illegal immigrant". These are culture-warry for obvious reasons; I'm mostly looking for engagement on the meta-level.