One percent of the world’s AI compute (LLM-grade GPU capacity) is in the UAE which does not have an AI Security Institute. I’ve planned to spend 6-9% of my bandwidth this month (2-3 days during May 2025) on encouraging the UAE to establish an AISI. Today is the first day.

However in my view even the most optimistic impact estimate of the successful execution of that plan doesn’t realistically lead to a greater than 2% shift in the prediction market of the UAE starting an AI Security Institute before 2026. Even if a UAE AISI existed, then it would not be allocated more than 1% to 5% (mode 2%) of the overall national AI budget (roughly $2b). Taking 2% of 2% of $2b is a maximum valuation of $800k for the entire project (I think that the median valuation would be significantly lower, but I’m not using the maximum here to be generous, but because I believe that — for this system — using the max value is more informative for decision making and easier to estimate than evaluating the 95th or 99.9th percentile value).

I was talking about this with my dad earlier, whose take was that attending the 1 day https://govaisummitmea.com/ on May 13th would be less than 0.01% of the work involved in actually pulling this thing off. My understanding of what he meant in more formal terms is that if your goal is for the UAE to have an AISI before 2026, and you then decompose each step of the plan to achieve that outcome into players in a Shapley value calculation, then acquiring these tickets has an average marginal contribution of at most 0.0001 times at most $800k, which is $80. And it would be foolish to pay the cost of one day of my time as well as tickets for me plus a collaborator when the return on that investment is — by this model — capped at $80.

Although my dad’s take here is reasonable given the information available to him, he doesn’t have the full context and the time he can allocate to learning about this project is restricted. Even though he is burdened by the demands of his various responsibilities, I’m grateful that supporting me in particular is one that he has always prioritized. I love Abba!

Here’s why Abba is wrong. There are 400 total seats. The cost is USD 1299 per head, so 2600 to register two attendees. At this price range it makes sense for a company to reimburse the fee to send representatives if it profits from building relationships with UAE policymakers. These will mostly be orgs not working directly on reducing AI x-risk. Although having 0.5% of attendees be alignment researchers is unlikely to affect the overall course of discussions, it is a counterweight which balances against worlds where this minority group has zero participation in these conversations. I think it may be as much as 3.25% of the work needed, which is ten times more than the floor break even point of 0.325% (2600/800k). But besides that, my team has been approved for up to 5 seats, we have a 10-page draft policy memo coauthored with faculty at https://mbrsg.ae/, and we can just ask for the registration fee to be waived. I agree that it would be insane to pay the full amount out of pocket. (Edit: we got the waiver request approved.)

Here’s why Abba is right. This post was written after midnight. Later today I will go to the Middle East and Africa’s largest cybersecurity event, https://gisec.ae/. I look forward to further comments and reviews.

Thread: Research Chat with Canadian AI Safety Institute Leadership

I’m scheduled to meet https://cifar.ca/bios/elissa-strome/ from Canada’s AISI for 30 mins on Jan 14 at the CIFAR office in MaRS.

My plan is to share alignment/interp research I’m excited about, then mention upcoming AI safety orgs and fellowships which may be good to invest in or collaborate with.

So far, I’ve asked for feedback and advice in a few Slack channels. I thought it may be valuable to get public comments or questions from people here as well.

Previously, Canada invested $240m into a capabilities startup: https://www.canada.ca/en/department-finance/news/2024/12/deputy-prime-minister-announces-240-million-for-cohere-to-scale-up-ai-compute-capacity.html. If your org has some presence in Toronto or Montreal, I’d love to have permission to give it a shoutout!

Elissa is the lady on the left in the second image from this article: https://cifar.ca/cifarnews/2024/12/12/nicolas-papernot-and-catherine-regis-appointed-co-directors-of-the-caisi-research-program-at-cifar/.

My input is of negligible weight, so wish to coordinate messaging with others.

One thing I like to do on a new LLM release is the "tea" test. Where you just say "tea" over and over again and see how the model responds.

ChatGPT-4 will ask you to clarify and then shorten its response each round converging to: "Tea types: white, green, oolong, black, pu-erh, yellow. Source: Camellia sinensis."

Claude 3 Opus instead tells you interesting facts about tea and mental health, production process, examples in literature and popular culture, etiquette around the world, innovation and trends in art and design.

GOODY-2 will talk about uncomfortable tea party conversations, excluding individuals who prefer coffee or do not consume tea, historical injustices, societal pressure to conform to tea-drinking norms.

Gemma-7b gives "a steaming cup of actionable tips" on brewing the perfect cuppa, along with additional resources, then starts reviewing its own tips.

Llama-2-70b will immediately mode collapse on repeating a list of 10 answers.

Mixtral-8x7b tells you about tea varieties to try from around the world, and then gets stuck in a cycle talking about history and culture and health benefits and tips and guidelines to follow when preparing it.

Gemini Advanced gives one message with images "What is Tea? -> Popular Types of Tea -> Tea and Health" and repeats itself with the same response if you say "tea" for six rounds, but after the sixth round it diverges "The Fascinating World of Tea -> How Would You Like to Explore Tea Further?" and then "Tea: More Than Just a Drink -> How to Make This Interactive" and then "The Sensory Experience of Tea -> Exploration Idea:" and then "Tea Beyond the Cup -> Let's Pick a Project". It really wants you to do a project for some reason. It takes a short digression into tea  philosophy and storytelling and chemistry and promises to prepare a slide deck for a Canva presentation on Japanese tea on Wednesday followed by a gong cha mindfulness brainstorm on Thursday at 2-4 PM EST and then keeps a journal for tea experiments and also gives you a list of instagram hashtags and a music playlist.

Probably in the future I expect if you say "tea" to a SOTA AI, it will result in a delivery of tea physically showing at up your doorstep or being prepared in a pot, or if there's more situational awareness for the model to get frustrated and change the subject.

The main thing we care about is consistency and honesty. To maximize that, we need to retrieve information from the web (though this has risks), https://openai.com/research/webgpt#fn-4, select the best of multiple summary candidates https://arxiv.org/pdf/2208.14271.pdf, generate critiques https://arxiv.org/abs/2206.05802, run automated tests https://arxiv.org/abs/2207.10397, validate logic https://arxiv.org/abs/2212.03827, follow rules https://www.pnas.org/doi/10.1073/pnas.2106028118, use interpretable abstractions https://arxiv.org/abs/2110.01839, avoid taking shortcuts https://arxiv.org/pdf/2210.10749.pdf, and apply decoding constraints https://arxiv.org/pdf/2209.07800.pdf.