I recently came to an important macrostrategy realization.
A “crux” or “crucial consideration” (used interchangeably) is some factor which determines a significant portion of how valuable our future is. A classic example is the importance of preventing human extinction, which could clearly be one of the most important factors determining how valuable our future is.
One purpose of macrostrategy research is to unearth such cruxes and understand how they fit together so that we can develop a robust strategy to maximize the value of our future.
Two Fundamental Types of Cruxes
What I realized is that there are fundamentally two different kinds of cruxes or crucial considerations, “primary cruxes/crucial considerations” (PCs/PCCs) and “secondary cruxes/crucial considerations” (SCs/SCCs).[1]
PCs are those cruxes which, taken together, if we get them right, are sufficient to get all SCs correct. SCs are those cruxes which, if we “punt” on them, that is to say, if we do nothing at all on them and focus instead exclusively on PCs; we can still get them right as long as we get the PCs right.
Preventing existential catastrophes means avoiding any events that drastically reduce humanity’s long-term potential, such as human extinction or a global AI-enabled stable totalitarian dictatorship.
Deep reflection is a state in which humanity has comprehensively reflected on all cruxes and determined a strategy which will maximize its future expected value (I call this "comprehensive reflection"), and is likely to act on that strategy. It’s closely related to concepts such as long reflection and coherent extrapolated volition.
Essentially, if we avoid anything that significantly reduces our potential, and then figure out how to maximize our potential, and simultaneously put ourselves in a position where we are likely to act on this knowledge, then we are likely to achieve our potential—which includes getting all of the SCs correct.
When we get a bit more granular and look at sub-cruxes within these two high-level super-cruxes, things get quite a bit more mushy. There are many primary sub-cruxes within these primary super-cruxes, but there are potentially multiple different sets of primary cruxes which could be sufficient for achieving humanity’s potential. Hence, while there may be multiple sets of primary cruxes that are sufficient, it is not clear whether there are any primary cruxes (except perhaps avoiding existential catastrophes) which are strictly necessary.
For example, if we could sufficiently slow down AI progress or not build superintelligent AI at all, at least not until we can do so safely, this could be a perfectly good solution to the problem of AI existential risk, if achievable.
On the other hand, if we were able to reliably and robustly use near human level AI to align and/or control slightly beyond human level AI, and do this carefully and safely such that we keep the AI aligned and/or under control, then continue scaling this up to increasingly intelligent AI, and if this worked in the way that optimists hope it will, then this could be a second perfectly good path to preventing AI existential risk.
(To be clear, I’m not taking a stance on whether either of these are achievable, I’m just using this to illustrate the point that “slowing down AI could prevent AI existential risk” and “scalable oversight could prevent AI existential risk” could both, in theory, simultaneously be true.)
A Sufficient Set Would Solve Macrostrategy
We might say that we have discovered a “sufficient set” of PCs if a thorough understanding of that set of PCs is enough to guarantee we get all SCs correct. Equivalently, you might say that a sufficient set of PCs is a set which, if fully understood, produces a strategy that allows us to maximize the value of the future. To discover a sufficient set of PCs would essentially be to solve macrostrategy.[2]
That said, you could argue that finding a sufficient set of PCs is not the best macrostrategy strategy. You might argue that it may actually be more efficient to solve many SCs individually, for example because you believe achieving deep reflection is unrealistic.
Or you might argue that trying to maximize value is the wrong goal altogether, and that we should instead focus exclusively on preventing existential risk, and only once we achieve that should we start focusing on increasing future value.
I think these are valid concerns, and explore this much more in my research on deep reflection. A few brief counter-concerns:
There may be far too many secondary crucial considerations for us to get all of them right without having some ambitious coherent over-arching process for getting all of them right
We may not get deep reflection by default before hard-to-reverse path dependencies such as settling space/deciding superintelligence governance, as most humans may not have much interest or patience with this
It may be increasingly difficult to shift our path toward deep reflection if we do not do so soon, especially as AI accelerates things
Frankly, I believe that a sufficient set of cruxes is much more within reach than many might at first assume. Very difficult perhaps, but achievable. One of my goals of creating the new macrostrategy forum “Crux” is to help surface primary cruxes and maybe even find a sufficient set, or at least help build the scaffolding and data for an AI that could automate macrostrategy research and help us find a sufficient set.
Crux/Crucial Consideration Type Disentanglement Glossary:
Instrumental cruxes (ICs): cruxes which are important to get right because by doing so we are likely to get terminal cruxes correct
E.g.: the order in which important technologies are created
Terminal cruxes (TCs): cruxes which are important to get right in-and-of-themselves because their status directly and intrinsically determines how much value there is in the universe
E.g.: the amount and intensity of suffering in the universe
Meta-cruxes (MCs): cruxes which directly help us solve other cruxes
E.g.: macrostrategy research processes, Crux forum, automated macrostrategy research
Super-crux: a crux which contains sub-cruxes, and to be clear, a crux can be both a sub-crux and a super-crux, almost all cruxes probably have many sub-cruxes and super-cruxes
E.g.: (In reference to the below sub-crux) deep reflection would help us maximize future value
Sub-crux: a crux within a crux; some factor which determines a significant portion of how valuable a crux is
E.g.: (In reference to the above super-crux) if moral realism is true, something like a long reflection might be the right version of deep reflection, if something like anti-moral-realism is true, it’s more likely to be something like coherent extrapolated volition
In previous work I used a somewhat related term “robust viatopia proxy targets” which referred to conditions that are important to achieve in order to be likely to converge on highly valuable futures.
Will MacAskill also recently introduced a term “societal primary goods” which I believe is also closely related, if interested in this topic I would very highly recommend his post "What sort of post-super intelligence society should we aim for" where he introduces this term and gives some compelling potential examples.
Of course, epistemically, we can never be 100% certain any plan will work, and with quantum uncertainty no plan ever can work with 100% certainty; so a "sufficient set" may be considered more of a useful fiction macrostrategy could aspire toward, rather than some concrete epistemic or definite reality.
Cruxes Decide Future Value
I recently came to an important macrostrategy realization.
A “crux” or “crucial consideration” (used interchangeably) is some factor which determines a significant portion of how valuable our future is. A classic example is the importance of preventing human extinction, which could clearly be one of the most important factors determining how valuable our future is.
One purpose of macrostrategy research is to unearth such cruxes and understand how they fit together so that we can develop a robust strategy to maximize the value of our future.
Two Fundamental Types of Cruxes
What I realized is that there are fundamentally two different kinds of cruxes or crucial considerations, “primary cruxes/crucial considerations” (PCs/PCCs) and “secondary cruxes/crucial considerations” (SCs/SCCs). [1]
PCs are those cruxes which, taken together, if we get them right, are sufficient to get all SCs correct. SCs are those cruxes which, if we “punt” on them, that is to say, if we do nothing at all on them and focus instead exclusively on PCs; we can still get them right as long as we get the PCs right.
At the highest level you might say that there are really only two primary cruxes, preventing existential catastrophes and achieving deep reflection.
Preventing existential catastrophes means avoiding any events that drastically reduce humanity’s long-term potential, such as human extinction or a global AI-enabled stable totalitarian dictatorship.
Deep reflection is a state in which humanity has comprehensively reflected on all cruxes and determined a strategy which will maximize its future expected value (I call this "comprehensive reflection"), and is likely to act on that strategy. It’s closely related to concepts such as long reflection and coherent extrapolated volition.
Essentially, if we avoid anything that significantly reduces our potential, and then figure out how to maximize our potential, and simultaneously put ourselves in a position where we are likely to act on this knowledge, then we are likely to achieve our potential—which includes getting all of the SCs correct.
When we get a bit more granular and look at sub-cruxes within these two high-level super-cruxes, things get quite a bit more mushy. There are many primary sub-cruxes within these primary super-cruxes, but there are potentially multiple different sets of primary cruxes which could be sufficient for achieving humanity’s potential. Hence, while there may be multiple sets of primary cruxes that are sufficient, it is not clear whether there are any primary cruxes (except perhaps avoiding existential catastrophes) which are strictly necessary.
For example, if we could sufficiently slow down AI progress or not build superintelligent AI at all, at least not until we can do so safely, this could be a perfectly good solution to the problem of AI existential risk, if achievable.
On the other hand, if we were able to reliably and robustly use near human level AI to align and/or control slightly beyond human level AI, and do this carefully and safely such that we keep the AI aligned and/or under control, then continue scaling this up to increasingly intelligent AI, and if this worked in the way that optimists hope it will, then this could be a second perfectly good path to preventing AI existential risk.
(To be clear, I’m not taking a stance on whether either of these are achievable, I’m just using this to illustrate the point that “slowing down AI could prevent AI existential risk” and “scalable oversight could prevent AI existential risk” could both, in theory, simultaneously be true.)
A Sufficient Set Would Solve Macrostrategy
We might say that we have discovered a “sufficient set” of PCs if a thorough understanding of that set of PCs is enough to guarantee we get all SCs correct. Equivalently, you might say that a sufficient set of PCs is a set which, if fully understood, produces a strategy that allows us to maximize the value of the future. To discover a sufficient set of PCs would essentially be to solve macrostrategy.[2]
That said, you could argue that finding a sufficient set of PCs is not the best macrostrategy strategy. You might argue that it may actually be more efficient to solve many SCs individually, for example because you believe achieving deep reflection is unrealistic.
Or you might argue that trying to maximize value is the wrong goal altogether, and that we should instead focus exclusively on preventing existential risk, and only once we achieve that should we start focusing on increasing future value.
I think these are valid concerns, and explore this much more in my research on deep reflection. A few brief counter-concerns:
Frankly, I believe that a sufficient set of cruxes is much more within reach than many might at first assume. Very difficult perhaps, but achievable. One of my goals of creating the new macrostrategy forum “Crux” is to help surface primary cruxes and maybe even find a sufficient set, or at least help build the scaffolding and data for an AI that could automate macrostrategy research and help us find a sufficient set.
Crux/Crucial Consideration Type Disentanglement Glossary:
Instrumental cruxes (ICs): cruxes which are important to get right because by doing so we are likely to get terminal cruxes correct
E.g.: the order in which important technologies are created
Terminal cruxes (TCs): cruxes which are important to get right in-and-of-themselves because their status directly and intrinsically determines how much value there is in the universe
E.g.: the amount and intensity of suffering in the universe
Meta-cruxes (MCs): cruxes which directly help us solve other cruxes
E.g.: macrostrategy research processes, Crux forum, automated macrostrategy research
Super-crux: a crux which contains sub-cruxes, and to be clear, a crux can be both a sub-crux and a super-crux, almost all cruxes probably have many sub-cruxes and super-cruxes
E.g.: (In reference to the below sub-crux) deep reflection would help us maximize future value
Sub-crux: a crux within a crux; some factor which determines a significant portion of how valuable a crux is
E.g.: (In reference to the above super-crux) if moral realism is true, something like a long reflection might be the right version of deep reflection, if something like anti-moral-realism is true, it’s more likely to be something like coherent extrapolated volition
Cross-posted to cruxforum.com
YouTube version:
In previous work I used a somewhat related term “robust viatopia proxy targets” which referred to conditions that are important to achieve in order to be likely to converge on highly valuable futures.
Will MacAskill also recently introduced a term “societal primary goods” which I believe is also closely related, if interested in this topic I would very highly recommend his post "What sort of post-super intelligence society should we aim for" where he introduces this term and gives some compelling potential examples.
Of course, epistemically, we can never be 100% certain any plan will work, and with quantum uncertainty no plan ever can work with 100% certainty; so a "sufficient set" may be considered more of a useful fiction macrostrategy could aspire toward, rather than some concrete epistemic or definite reality.