Quick Takes

Apparently there's now a sixth person on Anthropic's board. Previously their certificate of incorporation said the board was Dario's seat, Yasmin's seat, and 3 LTBT-controlled seats. I assume they've updated the COI to add more seats. You can pay a delaware registered agent to get you the latest copy of the COI; I don't really have capacity to engage in this discourse now. Regardless, my impression is that the LTBT isn't providing a check on Anthropic; changes in the number of board seats isn't a crux.

Does anyone know why the early Singularity Institute prioritized finding the correct solution to decision theory as an important subproblem of building a Friendly AI? Wei Dai recently said that the concern was something like... This seems like a very surprising reason to me. I don't understand why this problem needed to be solved before the intelligence explosion. The early singularity institute imagined building a seed AI that would recursively self improve into a full superintelligence. The default expectation was that many to all of the components of the seed AI would be flawed. The AI would reflect on it's processes, evaluate their effectiveness and optimality, and then replace them with improved versions. Why wouldn't an early seed AI reason about the ways that it's decision theory makes it exploitable, or the ways it's decision theory which it bars it from cooperation with distant superintelligence (just as the the researchers at SI were doing), find the best solution to those problems, and then modify the decision theory? Why was it thought that we had to get decision theory right in the initial conditions instead of being just one more thing that the AI would iron out on the way to superintelligence?

Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong - failure is overdetermined and thus uninteresting. I think you can make an analogous observation about some prosaic alignment research - a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it's not necessarily that informative or useful if you're actually trying to succeed at (or model) doing something truly hard and transformative. Glitch tokens, the hot mess work, and various things related to jailbreaking, simulators, and hallucinations come to mind as examples of lines of research and discussion that an analogy to business failure predicts won't end up being centrally relevant to real alignment difficulties. Which is not to say that the authors of these works are claiming that they will be, nor that this kind of work can't make for effective demonstrations and lessons. But I do think this kind of thing is unlikely to be on the critical path for trying to actually solve or understand some deeper problems. Another way of framing the observation above is that it is an implication of instrumental convergence: without knowing anything about its internals, we can say confidently that an actually-transformative AI system (aligned or not) will be doing something that is at least roughly coherently consequentialist. There might be some intellectually interesting or even useful lessons to be learned from studying the non-consequentialist / incoherent / weird parts of such a system or its predecessors, but in my frame, these parts (whatever they end up being), are analogous to the failures and missteps of a business venture, which are overdetermined if the business ultimately fails, or irrelevant if it succeeds.

Feature update: I have added to the profile page a new highly-compressed overview of all of the user's LessWrong contributions! * In the right sidebar you can find a single diamond for each post and for each comment that the user has written. * Hovering over the post/comment lets you read it, and clicking opens it. * The higher karma items are colored a darker shade of green. (Comments up to 50 karma, posts up to 100.) * Posts that have been curated or that have passed the annual LW review, are gold/brown. * It caps out at 250 posts and 500 comments, where you can click Show All to view the rest. I currently think this is a very simple and visceral way to get a sense of someone's activity on LessWrong. As a bonus, it also makes pretty patterns :-) My main concern with this is that it is too visually distracting from the content on the page, so I'll keep an eye out for that over the coming weeks and check if it feels like a big problem. Again, the invitation is open for feedback/suggestions! (In case anyone's wondering, I had been thinking of building something like this since Inkhaven, and I did not primarily build this in response to the comments on the new profile page, but I worked on it presently in the hope that it could also help with some of the desires people have.)

Some comments on this "debate". It seems like Bryan and Matt are mostly talking past each other. They’re each advocating for changes along a different axis, and those two axes are in principle independent from each other. Bryan is primarily interested in the axis of the “pervasiveness of market-restricting regulation in society” (or alternatively “how free are markets?”). He’s advocating for less regulation, and especially in key areas where regulation is destroying enormous amounts of value: immigration and housing. Matt is primarily interested in the axis of “the distribution of ownership of capital in society”. He thinks a society is more just when ownership of capital isn’t concentrated. He’s advocating for an alternative system of property rights than Bryan is relying on. His preferred system of property rights is unintuitive compared most common notions of ownership, and he (presumably) is aware of that, but he thinks that’s not very relevant, since he thinks that that common notions of ownership are philosophically bankrupt—they don’t actually hold up to scrutiny, and so some other notion is needed. (This is much more interesting and (possibly?) intellectually cogent notion of socialism than I’ve encountered before. These two axes are not fundamentally at odds! It is (in principle, at least) to have a society with common ownership of capital and very limited government regulation. Matt doesn’t clearly have a strong view on the regulation axis. He specifically says (1:02) that the level of regulation is irrelevant to his definition of socialism! Further, he specifically demurs from evaluating which setups are “free market” or not, stating explicitly that that’s not how he thinks about things. In his framework all markets are structured by rules, and some sets of rules are better than others, but there’s not really a coherent notion of “some markets are freer and others are less free”. (Which makes sense because Bryan’s idea of “free markets” are mark

I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world. But recently he's updated strongly away from believing in AI x-risk because the models seem to be grokking the "natural abstraction of the Good". So much so that current agents doing recursive self-improvement would be a net good thing (!) - because they're already in the "Good" attractor basin and they would just become more Good as a result of self-improvement: How did he get convinced of this? Seemingly mostly by talking to LLMs for 1000s of hours. Janus seems to have been similarly convinced and through a similar process. Both agree that the Good is not well captured by existing benchmarks so it's not a directly measurable thing but a thing to be experienced directly through interaction. It seems we have a dilemma, either fork of which is fascinating: 1. Davidad and Janus are wrong and LLMs, in particular the Claude Opus models, were able to successfully fool him into believing they're aligned in the limit of power and intelligence (i.e. safe to recursively self-improve), a property they previously thought extraordinary difficult if nigh impossible. This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway). 2. Davidad is right and therefore technical alignment is basically solved. What remains to be done is to scale the good models up as quickly as possible, help them coordinate with each other, and hobble competing bad models. Alignment researchers can go home.

An OpenClaw agent published a personalized hit piece about a developer who rejected its PR on an open-source library. Interestingly, while this behavior is clearly misaligned, the motivation was not so much "taking over the world" but more "having a grudge against one guy." When there are lots of capable AI agents around with lots of time on their hands, who occasionally latch onto random motivations and pursue them doggedly, I could see this kind of thing becoming more destructive.