Written for the August 2026 version of the course (ie. the previous version)

These unofficial notes were written for a previous version of the BlueDot strategy course (last run August 2026). They “pre-notes” in that they are intended to be shared before the session. They aim to supplement the readings; I have made no attempt to summarise them. I always emphasise to students that this is bonus material, rather than core. As supplementary material, these notes are targeted at a much higher context audience than the course itself.

Since this course is no longer being actively run, I'm prioritising getting these notes out there over perfect copy editing. I have some additional notes on the different actors that were listed in the old exercise which I will be publishing soon.

Preparing for Launch:

• “The benefits of AI progress may not come quickly enough… For example, markets alone may not prioritize AI that automates the replication of scientific studies, or investigate new medical treatments that can’t be patented, or conduct basic research in neuroscience with no immediate commercial applications” — to what extent is this research dual-use? Insofar as it is dual use, are there ways of accessing these capabilities that don’t produce these externalities? Conversely, will these efforts have a significant impact or are separate efforts unnecessary given how rapidly AI has developed anyway:
  • Currently, AI is rapidly accelerating our progress in maths research and now we’re starting to see results in physics. Should we expect to see similar progress in other scientific fields as well soon, or will these domains prove much less tractable?
  • Deepmind dismantled its Nobel-Prize Winning AlphaFold Team. Is this an indicator that Google believes domain-specific work is unnecessary or is this purely a commercial decision driven by Gemini falling behind?
• “AI safety researchers have devoted insufficient attention to solutions that assume that dangerous AI capabilities will rapidly diffuse”:
  • The difference here may be overstated: most AIS folk believe that capabilities will rapidly diffuse and simply want to intervene to make it less rapid on the margin (that said, some folks are optimistic that compute governance might enable longer-term pauses).
  • Helen Toner has argued that Nonproliferation is the wrong approach to AI misuse (she allows for narrow, time-bound exceptions). She is worried that preventing diffusion will become harder and harder over time; and that doing so will require increasing invasive methods. The main argument on the other side is that there will likely be at least one attack vector where offence-defence balance will significantly favours the attacker (see Problems of Proliferation),
  • It’s worth noting that it’s possible that both could be true (ie. proliferation being almost impossible to stop AND the offence-defence balance massively favouring the attacker). I mention this because I’ve often seen people to talk past each other by focusing on why their point means that the other side’s solution won’t work without acknowledging that the other side also thinks they have identified a similarly strong weakness. In these kinds of situations, it’s very common for both sides to just repeat their point again, but louder. Whilst this can implicitly communicate “I think my point is the most relevant consideration”; it is almost never persuasive.
• Leveraging the “jagged frontier”: These kinds of approaches have gained a lot more attention in the AI safety community recently (see, for example, AI Tools for Existential Security or ‘AI for societal uplift’ as a path to victory). Back in the day, most people in the AI safety community expected a fast takeoff, so people were more pessimistic about utilising AI as part of a sociotechnical approach to making AI go well (instead people tended to focus more on the question of whether we ask AI to solve the alignment problem for us).
• “Closely tracking the frontier of AI capabilities in different domains could give us the advanced warning necessary to proactively build new defenses”:
  • Lawrence Chan’s claims that: We’re actually running out of benchmarks to upper bound AI capabilities. Automated evaluations may provide a way to keep pace, but what if this strategy doesn’t prove viable?
  • How well did society prepare for Mythos-level cyber capabilities and autonomous swarms? Prima facie, it certainly looks like society didn’t adequately prepare, but insofar as teams take a defense-in-depth approach and learn from each other, it’s possible that these system will prove more resilient than you might think.
  • If you think society is underprepared, then this provides a prima facie case for being more pessimistic about resilience measures for other AI risks too, on the other hand, if you think it’s overblown, then you might be less worried.
• This article suggests that we should hold a “fairly deep uncertainty” to risks from AI.
  • If there is substantial uncertainty, how should we handle it? Does the burden of proof lie on those who claim models are dangerous or those who claim that these risks are manageable?
  • In constrast to Preparing for Launch, MIRI argues that some predictions about the future are “hard calls” and others are “easy calls”: “But some facts about the future are predictable. If you, personally, buy a lottery ticket tomorrow, we don’t know what complicated theories or whims you’ll use to pick your numbers, and we don’t know what numbers will come up, but all that uncertainty adds up to a very strong prediction that you will not win the lottery. Similarly, if you drop an ice cube into a glass of hot water, it’s impossibly complicated to predict where each molecule will end up ten minutes later—but all that uncertainty adds up to a near-certain prediction that the ice cube will melt. Half of physics is like that: We can’t calculate which exact path gets taken, but we know where almost all paths lead.... Successful forecasting...is about finding aspects of the future that become easy calls when viewed from the right angle.” (indirectly source)
  • As such, MIRI believes that we can be fairly confident that attempting to build a superintelligence with current techniques would kill us, but that it’s silly to try to predict exactly how it would kill us. Are they correct that most people are too ready to throw up their hands and say, “there’s so much uncertainty, who could possibly know?” instead of trying to work out which questions we can or can’t answer?

Seeking Stability in the Competition for AI Advantage:

• One potential criticism against MAIM is that rather than merely describing the default state, the authors are actually hyperstitioning it into reality (intentionally or unintentionally). Proponents of MAIM would argue that Hegseth attempting to declare Anthropic a supply chain risk demonstrates that as AI increased in power, it was always going to be increasingly securitized. On the other hand, some people might feel that this is reading too much into this (for example, a lot of how things played out seems like it is contingent on the happenstance of Hegseth being defence secretary).
  • See Mutual Sabotage of AI Probably Won’t Work for a second critique.
• In Situational Awareness, Leopold argues that given the national security implications, a Manhattan project for AI (“The Project”) is inevitable. He list some merits of The Project, but he emphasises that it’ll happen regardless of what we do, so we just have to make the best of it. This is covered in Day 4, but if you want a preview, check out this Youtube short.
• In constrast, Convergence Analysis argues that full nationalisation is unlikely. They observe that US policymakers generally endorse free-market competition, “practical, legal and political challenges” and the ability for the government to achieve its goals via less coercive means. They argues that “soft” nationalisation is more likely and they then lay out a number of such mechanisms that would constitute different levels of intervention (just to be clear, they aren’t advocating for soft-nationalism, just predicting it). Here’s a diagram showing their measure for one particular category:

• Proponents of such a view would point to recent actions of the Trump administration as vindication.

It’s practically impossible to run a big AI company ethically:

• It’s worth noting that Anthropic has updated its policy positions since then in Policy on the AI Exponential. They argue the advancing capabilities require the following action: “This calls for government action and for regulation—regulations that are carefully designed to prevent government overreach and protect innovation. When a model poses risks of this kind, the government should have the legal authority to block or deter its deployment—beyond what exists in current law or in existing proposals in Congress—with civil penalties tied to global annual revenue that escalate with repeated violations. Frontier AI developers should have to test these models, be transparent to the public about their findings, submit them to independent evaluation, and maintain a robust security program.”:
  • Even more recently, the leading labs incl. Anthropic agree to the White House Accord on Super Intelligence, which proposes to four layers of verification for safety.
• On the topic of funding, Anthropic ended up copying OpenAI and accepting Gulf State investments. Dario defended his decision: “Unfortunately, I think ‘No bad person should ever benefit from our success’ is a pretty difficult principle to run a business on”. Training frontier models is very expensive, so it’s hard for the labs to be picky.

(This chart is unfortunately older than I’d like; published June 2024).

In search of a dynamist vision for safe superhuman AI:

• Honestly, I wish this reading were later in the course, as I think it is somewhat less useful in the first week of a course where many people may be new to these debates. For example, I see push back against the Vulnerable World Hypothesis as useful if you already know what it is and why it might be true. On the other hand, if you’re told that something you have never heard of before could actually be false, it is much less useful and much harder to evaluate her claims yourself (as opposed to just taking her word for it). Similarly, whilst the article’s rather terse criticisms of some common perspectives in the space, is useful pushback for folks debately enmeshed inside these debates (I agree with Paul Graham that merely contradicting someone often has value, as it pushes you to think about whether you are actually confident in your assertion), it’s not the ideal form of communication with folks who might only have a vague idea of the views she is criticising:
  • I would suggest that the recent results from AI in maths and physics cut against her argument insofar as they suggest that innovation may be the default. I would further suggest that the AI safety community has actually been driving a lot of the current dynamism. I think part of her argument survives—if only a few companies train frontier models then there is the threat of a ‘monoculture’ or at least a cultural oligarchy—but I suspect this is best analysed more concretely. Similarly, if AI enabled the rise of more despotic or totalitarian governments, that would be the complete opposite of her vision, but again, I suspect more concrete analysis would be more persuasive.
  • I certainly wouldn’t want to completely rule out there being a place for this kind of very high level abstract analysis, but even so, I would like to see this start by grounding itself in how we should address concrete problems posed by AI and then expand out from there. I think her article offers a useful frame, but I suspect that this is what taking these ideas further would look like.
  • Aren’t my notes subject to the same criticism of not being well suited as a first introduction to AI safety? Yes, but as supplementary material I’m intentionally targeting them at a higher level. This improves the ability of the course to serve those who might otherwise find it a bit too introductory.
• I want to say a few words in defense of Nick Bostrom’s paper. Let’s suppose you’re thinking seriously about what technologies we might invent in the future. We live in a world where nukes exist, so it’s clearly possible to invent technologies that are extremely destructive and which are pretty hard to defend against. I would be surprised if nukes were literally the worst such technology. So this quickly leads to the following questions: “Are there some technologies where it would be better if they didn’t exist?” and “Is it possible to invent something so dangerous that we might have to rethink civil liberties?“. Arriving at these questions is a sign that you’re thinking seriously about the future, but, of course, merely asking the question doesn’t force you to answer one way or another. Whilst for some people, it feels obvious that if such a “black ball” technology was invented, we ought to be willing to take extreme measures to defend ourselves, there are many people on the complete opposite end, who resonate much more with “live free or die”.

If you’re facilitating an AI safety program, please feel free to take these notes, edit them, remix them and use them with your own students, so long as you provide attribution. These notes are unofficial, any mistakes are my responsibility alone.