Quick Takes

The most possible way acausal trade turns out to be infeasible is probably like ascertaining that other party has decision procedure that is actually dependent on yours. And finding such parties, that would in turn look back at you. Like, let's say the basic case for the trade is like that. You want A to exist and have B-suitable opportunity. You can make A out of it, but at 0.5 efficiency. Someone else has this situation but reversed, they want B to exist and have A-suitable opportunity. You can trade and both will be better off. Of course even better would be the situation when you look like you are going do your end of the deal, but then not do it, you get 150% out of that. So, you should do your end of the deal only if that makes the other party to switch from not-A to A, with enough credence that gain outweighs the cost. But, how do you ascertain that? That sounds super difficult, and there incentives to be deceptive in the impression you give off. Okay, here is one way to look at this. Let's say there are large agents that you can't effectively simulate, because their policies depend on large fraction of compute you yourself have. But some of them were not always like that, some of them were small and then grew. You probably can exploit that? Like, you can look into them and see that they commit to become particular type of large agent. So, you can do actions, that their later self would be able to see, even if they can't now. Small visible agent commits to something -> You in mid stage see them and do your part -> Small agent becomes large and able too see you in mid stage, and verifies your execution, and does its part too. It's plausible a lot of this stuff demands very early commitments, and sprouts from that. Of course as you are small, you can make very bad choices, so you should do less choices. Unfortunately. Also something something successor alignment.

Here are some of my top candidates for big pushes to do right now on technical AI safety (low effort notes): * Much better model organisms / misalignment analogies: * Doing a wider set of pessimized training runs * Good candidate for lots of AI labor automation? Like maybe good to try to set up pipelines for building these envs. * Demonstrating risks from fitness-seekers/reward-seekers empirically * Even on current models with better tests, see here * Demonstrating various types of memetic spread of misalignment? * Actually do control * Build pipelines for red-teaming monitors and the agent itself. For the agent red-teaming, I'd put particular focus on checking whether it continues malign trajectories. * Scaffold integrated control features and other non-monitoring runtime control measures * Human response and auditing * Improving async and sync monitoring * Agent security features * Surveilling for rogue internal deployments (as in, building after-the-fact detection methods for rogue deployments) * Preparing for handoff and elicitation * Get AIs generically better at conceptual work * Have a plan for the evals we ultimately need to see if handoff/deference would go well and start iterating on earlier versions * These presumably will involve a bunch of manual scoring, so we'll need to build a process for it. * Analyze AI biases and epistemics and improve across many domains * Build the anti-slop/anti-mundane-misalignment coalition via doing ratings of AIs and applying some pressure to improve on these ratings. This could focus on a variety of related issues. * The hope is basically that there might be widespread interest in removing/redacting mundane misalignment and other non-misalignment behavioral problems that reduce productivity and large parts of this seem differentially good. So, if we could make this a salient metric, AI companies might improve this. A lot of the difficulty would be in measuring

The Mythos checkpoint that the AISI ran their cyber evals on was actually an earlier version of the model. The one released with project Glasswing is much stronger.

It's wacky that we don't know if current AIs would try to take over the world if: * Takeover was easy * Takeover was an obvious way to accomplish an otherwise extremely difficult task * Takeover didn't have clearly evil vibes while clearly being takeover * The situation didn't look like a test Checking is hard! ---------------------------------------- I suspect the answer is basically no, at least for AIs from Anthropic and OpenAI, but I'm not that confident. Models sometimes do crazy (misaligned) actions to try to succeed at tasks when they get stuck.

[Image: image.png]

Some musings on the shape of the world before complete AI takeover: Some of the AI companies/leaders - the ones who talk about basic income - seem to have a daydream according to which AI (possibly in combination with robotics) will put everyone out of a job, except for the people who work at the AI companies; but all those unemployed people will get a payment doled out by their government. If we assume, for the sake of discussion, a world in which there is still a US government, and still multiple AI companies, each of which now sits atop an AI/robot economy that uses its products... one may suppose an agreement in which a fraction of each company's income or resources is given to the government, which in turn gives it to the unemployed people so they can afford AI-economy goods and services. Someone might also become a customer/subscriber/dependent of a specific AI company and thereby get access to its subsidiary economy without going through the government. Finally, there may also be an "anti-AI" economy, among humans who agree to avoid AI products entirely. It's not quite clear what happens to countries that don't have their own AI industries, but they might try to be part of such arrangements. One has to suppose that the AI powers also have a treaty or protocol that sharply regulates proliferation of core AI technology to new countries. On top of the uncontrolled effects of AI on society, we could have a deliberate effort to create this kind of a world. However, one doesn't expect AI capabilities to stand still (unless AI treaties not only restrict AI development outside the trusted powers, but also restrict AI development within them), and so AI won't just produce entities fit to do white-collar work, it will produce entities fit to converse as equals with the new human aristocracy (by which I mean the remaining humans in charge of the AI companies and in charge of the government). To this picture of a managed AI future, one might therefore add a debate am

I continue to think Hantavirus becoming a pandemic this year is much less than 5%. There’s a chance the opportunity to make 10x gains on the prediction market leads a WHO insider to falsely characterize it as a pandemic in what technically counts as an “official communication.” Andes virus, the strain of hantavirus that provoked the prediction market, is endemic in two countries. It causes hundreds of cases per year. The strain that infected the Hondius appears to be wild type. In other words, unlike COVID-19, it doe not have new, recent mutations that make it unusually efficient at infecting humans compared to its historic baseline. If this virus had pandemic potential, I believe we’d have seen it causing the same kinds of mass outbreaks in local villages and towns that we saw in Wuhan at the start of COVID. Argentinian and Chilean villages have the kinds of rodents that spread it efficiently. If we aren’t getting mass outbreaks there, why think we’ll get mass outbreaks in regions where that rodent reservoir is absent? None of this means we shouldn’t take it seriously. A 40% CFR hantavirus pandemic would be one of the worst catastrophes in human history. I just think that outcome is highly unlikely. I’m not sure people on LessWrong entirely understand how Polymarket works on a mechanical level, and suspect it leads them to grossly misinterpret these markets. Currently, the “hantavirus pandemic 2026?” market is at 8.9 cents. To participate, you and counterparty jointly collateralize a new $1 share - in this case, ”no” pays 91.1 cents, “yes” pays 8.9 cents. Your capital is then locked up for the next 7 months waiting for the market to resolve. Whoever’s correct gets $1. If you believe the likelihood of a pandemic is 0.1% and want to sweep the order book to reflect that probability, it’ll currently cost you about $7M to collateralize all those shares. If correct, you stand to gain about $160k in 7 months - a measly 2% gain, far worse than you’d be likely to do b