This post is still in draft, so any feedback would be greatly appreciated 🙏. It'll be posted as a full, proper Less Wrong/EA Forum/Alignment forum post, as opposed to just a short-form, when it's ready 🌱🌿🌳.

 ✉️ PM via LW profile   

📲 QR Code

This post is a collaboration between Chris Leong (primary author) and Christopher Clay (editor), written in the voice of Chris Leong.

We have worked very hard on this^[3] and we hope you find it to be of some use. Despite this work, it will most likely contain enough flaws that we will be at least somewhat embarrassed at having written at least parts of it and yet at some point a post has to go out into world to fend for itself.

There's no guarantee that this post will have the kind of impact that we deeply wish for, or even any achieve anything at all... And yet - regardless^[4] - it is an honour to serve^[5] especially at this dark hour when AGI looms on the horizon and doubt has begun to creep into the hearts of men^[6] 🫡. The sheer scale of the problem feels overwhelming at times, and yet... we do what we can^[7].

This post^[8] doubles^[9] as both as serious attempt to produce an original "alignment proposal"^[10] and a passion project^[11]. The AI safety community has spend nearly two and a half decades trying to convince people to take AI risks seriously. Whilst there have been significant successes, these have also fallen short^[12]. Undoubtedly this is incredibly naive^[13], but perhaps passion and authenticity^[14] can succeed where arguments and logic have failed ✨🌠. 

Acknowledgements:

The initial draft of this post was produced during the 10th edition of AI Safety Camp. Thanks to my team: Matthew Hampton, Richard Kroon, and Chris Cooper, for their feedback. Unlike most other AI Safety Camp projects, which focus on a single, unitary project, we were more of a research collective with each person pursuing their own individual projects.

I am continuing development during the 2025 Summer Edition of the Cambridge ERA: AI Fellowship with an eye to eventually produce a more formal output. Thanks to my research manager, Peter Gebauer and my mentor, Prof. David Manley.

I also greatly appreciate feedback provided by many others^[15], including: Jonathan Kummerfeld.

We stand at a crucial moment in the history of our species. Fueled by technological progress, our power has grown so great that for the first time in humanity’s long history, we have the capacity to destroy ourselves—severing our entire future and everything we could become.

Yet humanity’s wisdom has grown only falteringly, if at all, and lags dangerously behind. Humanity lacks the maturity, coordination and foresight necessary to avoid making mistakes from which we could never recover. As the gap between our power and our wisdom grows, our future is subject to an ever-increasing level of risk. This situation is unsustainable. So over the next few centuries^[20], humanity will be tested: it will either act decisively to protect itself and its long-term potential, or, in all likelihood, this will be lost forever — Toby Ord, The Precipice^[21]

We have created a Star Wars civilization, with Stone Age emotions, medieval institutions, and godlike technology — Edward O. Wilson, The Social Conquest of Earth^[22]

Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct — Nick Bostrom, Founder of the Future of Humanity Institute, Superintelligence^[23]

If we continue to accumulate only power and not wisdom, we will surely destroy ourselves — Carl Sagan, Pale Blue Dot^[24]

Never has humanity had such power over itself, yet nothing ensures that it will be used wisely, particularly when we consider how it is currently being used…There is a tendency to believe that every increase in power means “an increase of ‘progress’ itself ”, an advance in “security, usefulness, welfare and vigour; …an assimilation of new values into the stream of culture”, as if reality, goodness and truth automatically flow from technological and economic power as such. — Pope Francis, Laudato si'^[25]

The fundamental test is how wisely we will guide this transformation – how we minimize the risks and maximize the potential for good — António Guterres, Secretary-General of the United Nations^[26]

Our future is a race between the growing power of our technology and the wisdom with which we use it. Let’s make sure that wisdom wins — Stephen Hawking, Brief Answers to the Big Questions^[27]

🎁 Additional quotes (from Life Itself) 

 ⬇️ I suggest initially skipping this section and focusing on the core argument for now^[33] ⬇️

Yes,  I've^[38] intentionally chosen a high bar for what I'll be arguing for in this post. I believe that there's a strong case to be made that the gameboard looks much less promising without them^[39][40].

Don't worry,  I'll be addressing a long list of objections^[41] ⚔️🌎. Let's go 🪂!

🚨💪🦾  —  1: This approach scales with increases in capabilities.
2) Forget marginal improvements, wisdom tech could provide the missing piece for another strategy, by allowing us to pursue it in a wiser manner.
3) Wisdom technology is likely favourable from a differential technology perspective. Many actors are only reckless because they're unwise.
4) Even a small coalition using these advisors to refine strategy, improve coordination and engage in non-manipulative persuasion could significantly shift the course of civilisation over time.
5) Suppose all else fails: training up a bunch of folks with both a deep understanding of the alignment problem and a strong understanding of wisdom seems incredibly useful.

Importance-Tractability-Neglectedness
 

This is a standard EA framework for considering cause areas. Wise AI is broad enough that I consider it reasonable to analyse it as a cause area. 

Safety-Freedom-Value

This framework is designed to appeal more to startup/open-source folks. These folks are more likely to put significant weight on the social value a technology provides and on freedom beyond mere utility.

Searching for Solutions
 

This framework is designed for folk who think current techniques are unlikely to work.

As I said, if you're serious, please  ✉️ PM me . If you think you might be serious, but need to talk it through, please reach out as well.

I'd be useful for me to know your background and how you think you could contribute. Maybe tell you a few facts about yourself, what interests you and drop a link to your LinkedIn profile?

If you scroll down, you'll see that I've answered some more questions about getting involved, but I'll include some useful links here as well:

•  List of potentially useful projects : for those who want to know what could be done concretely. Scroll down further for a list of project lists ⬇️.
•  Resources for founding an AI safety startup or non-profit : since I believe there should be multiple organisations pursuing this agenda
•  Resources for getting started in AI Safety more broadly : since some of these resources might be useful here as well

Sure. I've written at the top of this post ⬆️ why I often try to dodge this question initially. But given that you've made it this far, I'll share some thoughts. Just don't over-update on my views 😂.

Given how rapidly AI is developing, I suspect that we're unlikely to resolve the millennia-long philosophical debate about the true nature of wisdom before AGI is built. Therefore, I suggest that we instead sidestep this question by identifying specific capabilities related to wisdom that might useful for steering the world in a positive direction.

I'd suggest that examples might include: the ability to make wise strategic decisions, non-manipulation persuasion and the ability to find win-wins. I'll try to write up a longer list in the future.

Some of these capabilities will be more or less useful for steering the world in positive directions. On the other hand, negative externalities such as accelerating timelines or enabling malicious actors.

My goal is to figure out which capabilities to prioritise by balancing the benefits against the costs and then incorporating feasibility.

It may be the case that some people decide that wisdom requires different capabilities than those I end up deciding are important. As long as they pick a capability that isn't net-negative, I don't see that as bad. In fact, I see different people pursuing different understandings of wisdom as adding robustness to different world models.

So the question is: "Why do I want to break wisdom down into separate capabilities instead of choosing a unitary definition of wisdom and attempting to train that into an AI system?"

Firstly, I think the chance of us being able to steer the world towards a positive direction is much higher if we're able to combine multiple capabilities together, so it makes sense to have a handle for the broader project, in addition to handles for individual sub-projects. I believe that techniques designed for one capability will often carry over to other capabilities, as will the challenges, and having a larger handle makes it easier to make these connections. I also think there's a chance that these capabilities amplify each other (as per the final few paragraphs of  Imagining and Building Wise Machines ^[60] by Johnson, Bengio, Grossmann, et al).

Secondly, I believe we should be aiming to increase both human wisdom and AI wisdom simultaneously. In particular, I believe it's important to increase the wisdom of folks creating AI systems and that this will then prove useful for a wide variety of specific capabilities that we might wish to train.

Finally, I'm interested in investigating this frame as part of a more ambitious plan to solve alignment on a principled level. Instead of limiting the win condition to building an AI that always (competently) acts in line with human values, the wise AI Advisors frame broadens it such that the AI only needs to inspire humans to make the right decision. It's hard to know in advance whether this reframing will be any easier, but even if it doesn't help, I have a strong intuition that understanding why it doesn't help would shed light on the barriers to solving the core alignment problem.

Yep, I was focused on it before. I now see that goal as overly narrow. The goal is to produce wise AI Advisors via any means. I think that Imitation Learning is underrated, but there are lots of other approaches that are worth exploring as well.

Optimism isn't about burying your head in the sand and ignoring the massive challenges facing us. That's denialism. Optimism is about rolling up your sleeves and doing what needs to be done.

The nice thing about this proposal from an optimistic standpoint is that, assuming there is a way for AI to go well for humanity, then it seems natural to expect that there is some way to leverage AI to help us find it^[61].

Additionally, the argument for developing wise AI advisors isn't in any way contingent on a pessimistic view of the world. Even if you think AI is likely to go well by default, wise AI advisors could still be of great assistance for making things go even better. For example, facilitating negotiations between powers, navigating the safety-openness tradeoff and minimising any transitional issues.

Short timelines add force to the argument I've made above, but they aren't at all a necessary component^[62].

Even if AI will be developed over decades rather than years, there's still enough different challenges and key decisions that unaugmented human wisdom is unlikely to be sufficient.

In fact, my proposal might even work better over long timelines, as it provides more time for AI advisers to help steer the world in a positive direction^[63].

I'm extremely worried about the incentives created by a general chatbot product. The average user is low-context and this creates an incentive towards sychophancy 🤥.

I believe that a product aimed at providing advice for critically important decisions would provide better incentives, even it were created by the same company.

Furthermore, given the potential for short timelines, it seems extremely risky 🎰 to rely purely on the profit motive, especially since there is a much stronger profit motive to pursue capabilities 💰💰💰. A few months' delay could easily mean that a wise AI advisor isn't available for a crucial decision 🚌💨. Humanity has probably already missed having such advisors available to assist us during a number of key decision points 😭.

I expect both the benefits and the externalities to vary hugely by capability. I expect some to be positive, some to be negative and some to be extremely hard to determine. More work is required to figure out which capabilities are best from a differential technology development perspective.

I understand that this answer might be frustrating, but I think it's worth sharing these ideas even though i haven't yet had time to run this analysis. I have a  list of projects  that I hope will prove fairly robust, despite all the uncertainty.

Oscar Delany makes this argument in  Tentatively against making AIs 'wise'  (runner up in the  AI Impact Essay Competition ).

This will depend on your definition of wisdom.

I admit that this tends to be a problem with how I tend to conceive of wisdom, however I will note that  Imagining and Building Wise Machines  ( summary ) takes the opposite stance - the wisdom, conceived of as metacognition - can actually assist with explainability.

But let's assume that this is a problem. How much of a problem is it?

I suspect that this varies significantly by actor. There's a reasonable argument that public institutions shouldn't be using such tools for reasons of justice. However, these arguments have much less force when it comes to private actors.

Even for private actors, it makes sense to use more legible techniques as much as possible, but I don't think this will be sufficient for all decisions. In particular, I don't think objective reasoning is sufficient for navigating the key decisions facing society in the transition to advanced AI.

But I also want to push back against the non-verifiability. You can do things like run a pool of advisors and only take dramatic action if more than a certain proportion agree, plus you can do testing, even advanced things like latent adversarial testing. It's not as verifiable as we'd like, but it's not like we're completely helpless here.

There will also be ways to combine wise AI advisors with more legible systems.

Inner alignment is an extremely challenging issue. If only we had some... wise AI advisors to help us navigate this problem.

"But this doesn't solve the problem, this assumes that these advisors are aligned themselves": Indeed, that is a concern. However, I suspect that the wise AI advisors approach has less exposure to these kinds of risks as it allows us to achieve certain goals at a lower base model capability level.

• Firstly, wise people don't always have huge amounts of intellectual firepower. So I believe that we will be able to achieve a lot without necessarily using the most powerful models.
• Secondly, the approach of combined human-AI teams allows the humans to compensate for any weaknesses present in the AIs.

In summary, this approach might help in two ways: by reducing exposure and advising us on how to navigate the issue.

If you're serious, please  ✉️ PM me . If you think you might be serious, but need to talk it through, please reach out as well.

I'd be useful for me to know your background and how you think you could contribute. Maybe tell you a few facts about yourself, what interests you and drop a link to your LinkedIn profile?

If you scroll down, you'll see that I've answered some more questions about getting involved, but I'll include some useful links here as well:

•  List of potentially useful projects : for those who want to know what could be done concretely. Scroll down further for a list of project lists ⬇️.
•  Resources for founding an AI safety startup or non-profit : since I believe there should be multiple organisations pursuing this agenda
•  Resources for getting started in AI Safety more broadly : since some of these resources might be useful here as well

Yes, I created a list:  Potentially Useful Projects in Wise AI . It contains a variety of projects from ones that would be marginally helpful to incredibly ambitious moonshots.

What other project lists exist?

Here are some project lists (or larger resources containing a project list) for wise AI or related areas:

•  AI Impacts Essay Competition : Covers the automation of philosophy and wisdom.

•  Fellowship on AI for Human Reasoning - Future of Life Foundation : AI tools for coordination and epistemics

•  AI for Epistemics - Benjamin Todd  - He writes: "The ideal founding team would cover the bases of: (i) forecasting / decision-making expertise (ii) AI expertise (iii) product and entrepreneurial skills and (iv) knowledge of an initial user-type. Though bear in mind that if you have a gap in one of these areas now, you could probably fill it within a year" and then provides a list of projects.

•  Project ideas: Epistemics - Forethought  - focuses on improving epistemics in general, not just AI solutions.

See the  AI Safety & Entrepreneurship wiki page  for resources including articles, incubation programs, fiscal sponsorship and funding.

AI being incrementally wiser might still be the difference between making a correct or incorrect decision at a key point.

Often several incremental improvements stack on top of each other, leading to a major advance.

Further, we can ask AI advisors to advise us on more ambitious projects to train wise AI. And the wiser our initial AI advisors are, the more likely this is to go well. That is, improvements that are initially incremental might be able to be leveraged to gain further improvements.

In the best case, this might kick off a (strong) positive feedback cycle (aka wisdom explosion).

Possibly 🤷.

I'm not convinced it's harder than any of the other ambitious alignment agendas and we won't know how far we can go without giving it a serious effort. Is training an AI to be wise really harder than aligning it? If anything, it seems like a less stringent requirement.

Compare:
• Ambitious mechanistic interpretability aims to perfectly understand how a neural network works at the level of individual weights
• Agent foundations attempting to truly understand what concepts like agency, optimisation, decisions are values are at a fundamental level
• Davidad's Open Agency architecture attempting train AI's that come with proof certificates that an AI has less than a certain probability of having unwanted side-effects

Is it obvious that any of these are easier than training a truly wise AI advisor? I can't answer for you, but it isn't obvious to me.

Given the stakes, I think it is worth pursuing ambitious agendas anyway. Even if you think timelines are short, it's hard to justify holding a probability approaching 100%, so it makes sense for folk to be pursuing plans on different timelines.

This section provides a high-level overview of some of my more speculative ideas. Whilst these ideas are very far from fully developed, I nonetheless thought it was worthwhile sharing an early version of them. That said, I'm very much on the set of "let a hundred flowers bloom". Whilst I think the approach I propose is promising, I also think it's much more likely than not that someone else comes up with an even better idea.

I believe that imitation learning is greatly underrated. I have a strong intuition that using the standard ML approach for training wisdom will fail because of the traditional Goodhart's law reasons, except that it'll be worse because wisdom is such a fuzzy thing (Who the hell knows what wisdom really is?).

I feel that training a deeply wise AI system requires an objective function that we can optimise hard on. This naturally draws me towards imitation learning. It has its flaws, but it certainly seems like we could optimise much harder on an imitation function than with attempting to train wisdom directly.

Now, imitation learning is often thought of as weak and there's certainly some truth in this when we're just talking about an initial imitation model. However, this isn't a cap on the power of imitation learning, only a floor. There's nothing stopping us from training a bunch of such models and using amplification techniques such as debate, trees of agents or even iterated distillation and amplification. I expect there to be many other such techniques for amplifying your initial imitation models.

See  An Overview of "Obvious" Approaches to Training Wise AI Advisors  for more information. This post was the first half of my entry into the AI Impacts Essay Competition on the Automation of Wisdom and Philosophy which placed third.

I think it's important to consider whether you might not be the right person to make progress on this.

I mean — wisdom? Who the hell knows what that is?

I expect most people who try to gain traction on this to just end up confusing themselves and others. So, you probably shouldn't work on this unless you're a particularly good fit.

On the other hand, there are very strong selection effects where those who are fools think they are wise and those who are wise doubt their own wisdom.

I wish I had a good answer here. All I can say is to avoid both arrogance and performative modesty when reflecting on this question.

There's a very simplistic version argument for this that proves too much: people have used this claim to argue that LLMs were never going to be able to learn to reason, whilst o3 seems to have conclusively disproven this. So I don't think that the standard version of this argument works.

It's also worth noting that in robotics, there are examples of zero-shot transfer to the real world. Even if wisdom isn't directly analogous, this suggests that large amounts of real-world experience might be less crucial than it appears at first glance.

All this said, I find it plausible that a more sophisticated version of this argument might post a greater challenge. Even if this were the case, I see no reason why we couldn't combine both embodied and non-embodied methods. So even if there was a more sophisticated version that ultimately turned out to be true, this still wouldn't demonstrated that research into non-embodied methods was pointless.

Sure 😊. Round 1️⃣2️⃣ 🔔, here we go 🪂!

As I said, if you're serious, please  ✉️ PM me . If you think you might be serious, but need to talk it through, please reach out as well.

I'd be useful for me to know your background and how you think you could contribute. Maybe tell you a few facts about yourself, what interests you and drop a link to your LinkedIn profile?

Useful link:

•  List of potentially useful projects : for those who want to know what could be done concretely. Scroll down further for a list of project lists ⬇️.
•  Resources for founding an AI safety startup or non-profit : since I believe there should be multiple organisations pursuing this agenda
•  Resources for getting started in AI Safety more broadly : since some of these resources might be useful here as well

Hi, I'm Chris (Leong). I've studied maths, computer science, philosophy and psychology. I've been interested in AI safety for nearly a decade and I've participated in quite a large number of related opportunities.

With regards to the intersection of AI safety and wisdom:

I won third prize in the  AI Impacts Competition on the Automation of Wisdom and Philosophy . It's divided into two parts:

•  An Overview of “Obvious” Approaches to Training Wise AI Advisors 

•  Some Preliminary Notes on the Promise of a Wisdom Explosion 

I recently ran an  AI Safety Camp project on Wise AI Advisors .

More recently, I was invited to continue my research as a ERA Cambridge Fellow in Technical AI Governance.

Feel free to connect with me on  LinkedIn  (ideally mentioning your interest in wise AI advisors so I know to accept) or  ✉️ PM me .  

Hi, I'm Christopher Clay. I was previously a Non-Trivial Fellow and I participated in the Global Challenges Project.

Yes, here are a few:

Imagining and Building Wise Machines: The centrality of AI metacognition 

 original paper   summary 

Authors: Yoshua Bengio, Igor Grossmann, Melanie Mitchell and Samuel Johnson^[78] et al

Wise metacognition can lead to a virtuous cycle in AI, just as it does in humans. We may not know precisely what form wise AI will take—but it must surely be preferable to folly.

The paper mostly focuses on wise AI agents, however, not exclusively so:

Prof. Grossmann (personal correspondence): "I like the idea of wise advisors. I don't think the argument in our paper is against it -it all depends on how humans will use the technology (and there are several papers on the role of metacognition for discerning when to rely on decision-aids/AI advisors, too)."

AI Impacts Automation of Wisdom and Philosophy Competition 

Organised by: Owen Cotton-Barrett
Judges: Andreas Stuhlmüller, Linh Chi Nguyen, Bradford Saad and David Manley^[79]

 competition announcement   Wise AI Wednesdays post   winners and judge's comments ^[80]

AI is likely to automate more and more categories of thinking with time.

By default, the direction the world goes in will be a result of the choices people make, and these choices will be informed by the best thinking available to them. People systematically make better, wiser choices when they understand more about issues, and when they are advised by deep and wise thinking.

Advanced AI will reshape the world, and create many new situations with potentially high-stakes decisions for people to make. To what degree people will understand these situations well enough to make wise choices remains to be seen. To some extent this will depend on how much good human thinking is devoted to these questions; but at some point it will probably depend crucially on how advanced, reliable, and widespread the automation of high-quality thinking about novel situations is^[81]

Beyond Artificial Intelligence (AI): Exploring Artificial Wisdom (AW)
Authors: Dilip V Jest, Sarah A Graham, Tanya T Nguyen, Colin A Depp, Ellen E Lee, Ho-Cheol Kim

 paper 

The ultimate goal of artificial intelligence (AI) is the development of technology that is best able to serve humanity and this will require advancements that go beyond the basic components of general intelligence. The term “intelligence” does not best represent the technological needs of advancing society, because intelligence alone does not guarantee well-being either for individuals or societies. It is not intelligence, but wisdom, that is associated with greater well-being, happiness, health, and perhaps even longevity of the individual and the society. Thus, the future need in technology is for artificial wisdom (AW)

I've discussed several possible counter-arguments above, but I'm going to hold off publicly posting about which ones seem strongest to me for now. I'm hoping this will increase diversity of thought and nerdsnipe people into helping red-team my proposal for me 🧑‍🔬👩‍🔬🎯. Sometimes with feedback there's a risk where if you say, "I'm particularly worried about these particular counter-arguments" you can redirect too much of the discussion onto those particular points, at the expense of other, perhaps stronger criticisms.

I've created a weekly post series on Less Wrong called  Wise AI Wednesdays .

Motivation

‘AI for societal uplift’ as a path to victory - LW Post: Examines the conditions in which a "societal uplift" - epistemics + co-ordination + institutional steering - might or might not lead to positive outcomes

N Stories of Impact for Wise AI Advisors - Draft 🏗️ : Different stories about how wise AI advisors could be useful for having a positive impact on the world.

Artificial Wisdom

Imagining and building wise machines: The centrality of AI metacognition by Johnson, Karimi, Bengio, et al. -   paper ,  summary : This paper argues that wisdom involves two kinds of strategies (task-level strategies & metacognitive strategies). Since current AI is pretty good at the former, they argue that we should pursue the latter as a path to increasing AI wisdom.

Finding the Wisdom to Build Safe AI by Gordon Seidoh Worley -  LW post : Seidoh talks about his own journey in becoming wiser through Zen and outlines a plan for building wise AI. In particular, he argues that it will be hard to produce wise AI without having a wise person to evaluate it.

Designing Artificial Wisdom: The Wise Workflow Research Organisation by Jordan Arel -  EA forum post : Jordan proposes mapping the workflows within an organisation that is researching a topic like AI safety or existential risk. AI could be used to automate or augment parts of their work. This proportion would increase over time, with the hope being that this would eventually allow us to fully bootstrap an artificially wise system.

Should we just be building more datasets? by Gabriel Recchia -  Substack : Argues that an underrated way of increasing the wisdom of AI systems would be building more datasets (whilst also acknowledging the risks).

Tentatively Against Making AIs 'wise' by Oscar Delany -  EA forum post : This articles argues that insofar as wisdom is conceived of as being more intuitive than carefully reasoned pursuing AI wisdom would be a mistake as we need AI reasoning to be transparent. I've included this because it seems valuable to have at least one critical article.

Neighbouring Areas of Research

What's Important In "AI for Epistemics"? by Lukas Finnveden -  Forethought : AI for Epistemics is a subtly different but overlapping area. It is close enough that this article is worth reading. It provides an overview of why you might want to work on this, heuristics for good interventions and concrete projects.

AI for AI Safety by Joe Carlsmith ( LW post ): Provides a strategic analysis of why AI for AI safety is important whether it's for making direct safety progress, evaluating risks, restraining capabilities or improving "backdrop capacity". Great diagrams.

AI Tools for Existential Security by Lizka Vaintrob and Owen Cotton-Barratt --  Forethought : Discusses how applications of AI can be used to reduce existential risks and suggests strategic implications.

Not Superintelligence: Supercoordination -  forum post ^[82]: This article suggests that software-mediated supercoordination could be beneficial for steering the world in positive directions, but also identifies the possibility of this ending up as a "horrorshow".

Human Wisdom

Stanford Encyclopedia of Philosophy Article on Wisdom by Sharon Ryan -  SEP article : SEP articles tend to be excellent, but also long and complicated. In contrast, this article maintains the excellence while being short and accessible.

Thirty Years of Psychology Wisdom Research: What We Know About the Correlates of an Ancient Concept by Dong, Weststrate and Fournier -  paper : Provides an excellent overview of how different groups within psychology view wisdom.

The Quest for Artificial Wisdom by Sevilla -  paper : This article outlines how wisdom is viewed in the Contemplative Sciences discipline. It has some discussion of how to apply this to AI, but much of this discussion seems outdated in light of the deep learning paradigm.

Applications to Governance

🏆 Wise AI support for government decision-making by Ashwin -  Substack  (Prize winning entry in the  AI Impacts Automation of Wisdom and Philosophy Competition ): This article convinced me that it isn't too early to start trying to engage the government on wise AI. In particular, Ashwin considers the example of automating the Delphi process. He argues that even though you might begin by automating parts of the process, over time you could expand beyond this, for example, by helping the organisers figure out what questions they should be asking.

Some of my own work:

🏆 My third prize-winning entry in the  AI Impacts Automation of Wisdom and Philosophy Competition  (split into two parts):

 Some Preliminary Notes on the Promise of a Wisdom Explosion : Defines a wisdom explosion as a recursive self-improvement feedback loop that enhances wisdom, unlike intelligence as per the more traditional intelligence explosion. Argues that wisdom tech is safer from a differential technology perspective.

An Overview of "Obvious" Approaches to Training Wise AI Advisors : Compares four different high-level approaches to training wise AI: direct training, imitation learning, attempting to understand what wisdom is at a deep principled level, the scattergun approach. One of the competition judges wrote: "I can imagine this being a handy resource to look at when thinking about how to train wisdom, both as a starting point, a refresher, and to double-check that one hasn’t forgotten anything important".

 Automating the Delphi Method for Safety Cases  — Philip Fox, Ketana Krishna, Tuneer Mondal, Ben Smith, Alejandro Tlaie  — Ashwin and Michaelah Gertz-Billingsley proposed automating the Delphi Mehod as a foot-in-the door for getting the government to use wise AI. Well, turns out these folk at Arcadia Impact have done it!

We stand at a crucial moment in the history of our species. Fueled by technological progress, our power has grown so great that for the first time in humanity’s long history, we have the capacity to destroy ourselves—severing our entire future and everything we could become.

Yet humanity’s wisdom has grown only falteringly, if at all, and lags dangerously behind. Humanity lacks the maturity, coordination and foresight necessary to avoid making mistakes from which we could never recover. As the gap between our power and our wisdom grows, our future is subject to an ever-increasing level of risk. This situation is unsustainable. So over the next few centuries, humanity will be tested: it will either act decisively to protect itself and its long-term potential, or, in all likelihood, this will be lost forever — Toby Ord, The Precipice

We have created a Star Wars civilization, with Stone Age emotions, medieval institutions, and godlike technology — Edward O. Wilson, The Social Conquest of Earth

Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct — Nick Bostrom, Founder of the Future of Humanity Institute, Superintelligence

If we continue to accumulate only power and not wisdom, we will surely destroy ourselves — Carl Sagan, Pale Blue Dot

Never has humanity had such power over itself, yet nothing ensures that it will be used wisely, particularly when we consider how it is currently being used…There is a tendency to believe that every increase in power means “an increase of ‘progress’ itself ”, an advance in “security, usefulness, welfare and vigour; …an assimilation of new values into the stream of culture”, as if reality, goodness and truth automatically flow from technological and economic power as such. — Pope Francis, Laudato si'

The fundamental test is how wisely we will guide this transformation – how we minimize the risks and maximize the potential for good — António Guterres, Secretary-General of the United Nations

Our future is a race between the growing power of our technology and the wisdom with which we use it. Let’s make sure that wisdom wins — Stephen Hawking, Brief Answers to the Big Questions

“It had taken Sutskever years to be able to put his finger on Altman’s pattern of behavior—how OpenAI’s CEO would tell him one thing, then say another and act as if the difference was an accident. ‘Oh, I must have misspoken,’ Altman would say. Sutskever felt that Altman was dishonest and causing chaos, which would be a problem for any CEO, but especially for one in charge of such potentially civilization-altering technology.”

The Optimist, Keach Hagey

Ilya Sutskever, once widely regarded as perhaps the most brilliant mind at OpenAI, voted in his capacity as a board member last November to remove Sam Altman as CEO. The move was unsuccessful, in part because Sutskever reportedly bowed to pressure from his colleagues and reversed his vote. After those fateful events, Sutskever disappeared from OpenAI’s offices so noticeably that memes began circulating online asking what had happened to him. Finally, in May, Sutskever announced he had stepped down from the company.

Time 100 AI 2024

Twitter

We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.

This way, we can scale in peace.

Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.

Safe Superintelligence Inc.