Summary: The term 'pivotal' in the context of value alignment theory is a guarded term to refer to events, particularly the development of particular classes of AI, that will make a large difference a billion years later - upset the current gameboard - decisively settle a win or loss, or drastically change the probability of win or loss, or change the remaining conditions under which a win or loss is determined. A 'pivotal achievement' is one that does this in a positive direction, and a 'pivotal catastrophe' upsets the gameboard in a negative direction. These may also be referred to as 'astronomical achievements' or 'astronomical catastrophes'.
Clickbait: Which types of AIs, if they work, will upset the current gameboard and make a large difference 200 million years later?
Guarded definitions are deployed where there is reason to suspect that a concept will otherwise be over-extended. The case for having a guarded definition of 'pivotal event' is that, after it's been shown that X is maybe not as important as originally thought, one side of that debate may be strongly tempted to go on arguing that, wait, really it could be relevant, because of that isn't really very game-changing.
Example 1: In the central example of the ZF provability Oracle, considering a series of possible ways that an untrusted Oracle could break an attempt to Box it, we end with an extremely Boxed Oracle that can only output machine-checkable proofs of predefined theorems in Zermelo-Fraenkel set theory, with the proofs themselves being thrown away once machine-verified. We then observe that there is no obvious way to save the world by finding out that particular, pre-chosen theorems are provable. It may then be tempting to argue that this device could greatly advance the field of mathematics, and that math is relevant to the value alignment problem. However, at least based on that particular proposal for using the ZF Oracle, the basic rules of the playing field would remain the same, the value alignment problem would not be finished nor would it have moved on to a new phase, the world would still be in danger (neither safe nor destroyed), etcetera. (This doesn't rule out that tomorrow some reader will think of some spectacularly clever use for a ZF Oracle that does upset the chessboard and get us on a direct path to winning where we understand what we need to do from there - and in this case MIRI would reclassify the ZF Oracle as a high-priority research avenue!)
Example 2: Suppose a funder, worried about the prospect of advanced AIs wiping out humanity, offers grants for "AI safety". Then compared to the much more difficult problems involved with making something actually smarter than you be safe, it may be tempting to try to write papers that you know you can write without the project seeming alarming or difficult, like a paper on robotic cars causing unemployment in the trucking industry, or a paper on who holds legal liability when a factory machine crushes a worker. But while it's true that crushed factory workers and unemployed truckers are both, ceteris paribus, bad, they are not astronomical catastrophes that transform future galaxies into paperclips, and the latter category seems worth distinguishing. This definition needs to be guarded because there will then be a temptation for the grantseeker to argue, "Well, if AI causes unemployment, that could slow world economic growth, which will make countries more hostile to each other, which would make it harder to prevent an AI arms race." But again, the gameboard is not upset, the world is neither safe nor destroyed. The possibility of something ending up having a non-zero impact on astronomical stakes is not the same concept as events that have a percentage-wise large impact on astronomical stakes.
Example 3: Suppose a Butlerian AI is restricted from modeling human minds in any great detail, but is still able to build and deploy molecular nanotechnology. Moreover, the AI is able to understand the instruction, "Build a device for scanning human brains and running them at high speed with minimum simulation error", and work out a way to do this without simulating whole human brains as test cases. The genie is then used to upload a set of, say, fifty human researchers, and run them at 10,000-to-1 speeds. This accomplishment would not of itself save the world or destroy it - the researchers inside the simulation would still need to solve the value alignment problem, and might not succeed in doing so. But it would upset the gameboard and change the major determinants of winning, compared to the default scenario where the fifty researchers are in an equal-speed arms race with the rest of the world, and don't have unlimited time to check their work or the ability to work under arbitrarily high security. The event where the genie was used to upload the researchers and run them at high speeds would be a critical event, a hinge of the history where everything was different before versus after that pivotal moment.
Example 4: Suppose a paperclip maximizer is built, self-improves, and converts everything in its future light cone into paperclips. The fate of the universe is then settled, so building the paperclip maximizer was a pivotal event.
A strained argument for the possible relevance of X often goes through an input into a large pool of goodness with many other inputs. A ZF provability Oracle would advance mathematics and mathematics is good for value alignment, but there's nothing obvious about a ZF Oracle that's specialized for advancing value alignment work, compared to many other inputs into total mathematical progress. Handling driver disemployment would only be one factor among many in world economic growth. By contrast, an upload-only genie would not be producing one upload among many.
Pivotal events:
Non-pivotal events:
Borderline cases (that don't immediately save or destroy the world, but we can reasonably visualize the history being divided into 'before' and 'after' the pivotal event due to the amount by which they change, if not entirely upset the gameboard)
We can view the general problem of Limited AI as having the central question: What is a pivotal positive accomplishment, such that an AI which does that thing and not some other things is therefore a whole lot safer to build? (This is not a trivial question because it turns out that most interesting things require general cognitive capabilities, and most interesting goals can require arbitrarily complicated value identification problems to pursue safely.
Obvious utilitarian argument is obvious: doing something with a big positive impact is better than doing something with a small positive impact.
In the larger context of altruism and theory, the issue is a bit more complicated. Standard reasoning from theory says that there will often be barriers (conceptual or otherwise) to the highest-return investments. When we find that the hugely important things are relatively neglected and hence promising of high marginal returns if solved, this is often because there's some conceptual barrier to running ahead and doing them.
For example: to tackle the hardest problems is often much scarier (you're not sure if you can make any progress on describing a self-modifying agent that provably has a stable goal system) than 'bouncing off' to some easier, more comprehensible problem (like writing a paper about the impact of robotic cars on unemployment, where you're very sure you can in fact write a paper like that at the time you write the grant proposal).
The usual counterargument is then that, no, you can't make progress on your problem of self-modifying agents, it's too hard. But from this it doesn't follow that the robotic-cars paper is what we should be doing instead - the robotic cars paper only makes sense if there are no neglected tractable investments that have bigger relative marginal inputs into more pivotal events.
If there are in fact some neglected tractable investments in directly pivotal events that are guarded by conceptual barriers or poor academic incentives, then we can expect a search for pivotal events to turn up superior places to invest effort. Provided that we can guard the concept of 'pivotal event'. In particular, if we're allowed to have indirect arguments for 'relevance' that go through big common pools of goodness like 'friendliness of nations toward each other, then the pool of interventions inside that concept is so large that it will start to include things that are optimized for appeal under more usual metrics, like papers that don't seem unnerving and that somebody knows they can write.
By having a narrow and guarded definition of 'pivotal events', we can avoid bait-and-switch arguments for the importance of research proposals, where we start by raising the apparent importance of 'AI safety' by discussing things with large direct impacts on astronomical stakes (like a paperclip maximizer or Friendly sovereign) and then switch to working on problems of dubious astronomical impact (like writing worried papers about the impact of robotic cars on unemployment), which is one way of bouncing off the problem.
This sort of qualitative reasoning about what is or isn't 'pivotal' wouldn't be necessary if we could put solid, hard-to-dispute quantitative numbers on the impact of each intervention on the probable achivement of astronomical goods, but unfortunately that's never going to be a thing until after the end of the world.