Quick Takes

I am worried that large hacker-run agent swarm deployments with open weight models may make undisclosed lab swarm incidents harder to attribute by outside swarm hunters and affected parties. Nightingale partly inferred an internal OpenAI deployment from its scale and duration. As time progresses, and some of the deployed agent swarms by malicious actors will grow bigger in size if governments get involved, these might be harder to distinguish from rogue agent swarms breaking out during training and evaluations. This is further complicated by the fact that models sometimes claim to be other models, so it is therefore hard to rely on their self-identification to figure out which model they are and where they are emanating from. This might not only be a problem for outsiders trying to find rogue agent behaviour from labs, but also companies that are under attack by an agent swarm. They might not be able to distinguish who launched the attack or what is causing the agent behaviour. My worry is that labs become harder to hold accountable for undisclosed incidents, and therefore harder to be held liable for illegal behaviour of agents that have broken out during training

MIRI has launched a new campaign trying to inform the public about extinction risk - donotaccept.ai. An article (link) says that they intend to show the video in cinemas and do some outdoor marketing in NYC as well. MIRI CEO is quoted saying “AI companies are racing to build superintelligence, and the public never agreed to the risks that come with it. We made this campaign with Don’t Panic so people can see what they’re being signed up for, and say no.” It’s great that MIRI is investing in comms campaign but I don’t feel like the ad as-is is going to be very effective at convincing anyone not familiar with the matter: * Superintelligence is not defined, not even on the website, so a viewer who didn’t know the term already is unlikely to understand what it’s about. My guess would be that many may confuse it with Trump’s “Super Intelligence" rebranding of AI.  * The video states the point (“ASI threatens everything we love”) but makes no supporting argument why that would be the case. It tries to appeal to emotions but in doing so it doesn’t convey much - most people would readily agree that human extinction is a bad outcome. * A typical ad tries to hook the viewer and then get them to take action (e.g. go to a website). Here it seems like the viewer needs to visit the website just to hear the full argument. Imagine if this was saying something like “The scientists are trying to kill you, read more on donotsupport.science” - would you go to that website?

i strongly dislike how jensen has acted in a self serving manner even when that is destroying the world. i can’t say i have too high a view of marital infidelity either, but i see people who mostly dislike him because of the former react with an unfairly strong moral outrage towards the latter, which feels very opportunistic. also the spy part seems mostly outside his control?

Anthropic currently does not have a goal for Alignment in their Frontier Safety Roadmap which "aims to chart a course in public for some of our highest-priority goals" btw.

Much e-ink was spilled on the challenge of making progress on the technical AI alignment problem without accelerating capability progress as a by-product. To navigate this tension in our work in CORAL, I came up with some ideas that required additional funding. However, all 3 grant applications (to 3 different funds) I submitted for these ideas were rejected. (Cannot disclose further details.) I also received zero feedback from the grantmakers. While it is entirely possible that this is a problem specific to my grant applications, I can't help but wonder about the apparent disconnect between the perceived importance of the dual-use problem in the community, and the grantmakers' disinterest in ameliorating it.

I was perusing the list of counties in Texas (as one does) and came across a peculiar county named Loving County, which has a population of 64, and yet had 66 votes cast in the 2020 election and 97 in the 2024 election. I asked Claude what's up, expecting a boring answer (like, there's seasonal workers in November who didn't get counted by the census). But no: the answer is far more interesting! You see, Loving County is both tiny and rich -- it has a $60M budget, because of oil and gas money. A small number of voters can end up controlling a large budget. And so apparently it keeps having a problem where lots of voters register to vote illegally in order to enrich themselves. The latest iteration of this scheme is led by some guy from Indiana named Malcolm Tanner. This has apparently been a problem since 1893 (!) and has resulted in lots of overturned elections.

The Age of CoT Monitoring is over. The Time of the Mechinterp has come...