There is an idea that I’ve sometimes heard around rationalist and EA circles, that goes something like “you shouldn’t ever feel safe, because nobody is actually ever safe”. I think there are at least two major variations of this:

1. You shouldn’t ever feel safe, because something bad could happen at any time. To think otherwise is an error of rationality.
2. You shouldn’t ever feel safe, because AI timelines might be short and we might be about to die soon.^[1] Thus, to think that you’re safe is making an error of rationality.

I’m going to argue against both of these. If you already feel like both of these are obviously wrong, you might not need the rest of this post.

Note that I only intend to dispute the intellectual argument that these are making. It’s possible to accept on an intellectual level that it would make sense to feel safe most of the time, but still not feel safe. That kind of emotional programming requires different kinds of tools to deal with. I’m mostly intending to say that if you feel safe, you don’t need to feel bad about that. You don’t need to make yourself feel unsafe; for most people, it’s perfectly rational to feel safe.

I do expect some of the potential readers of this post to live in a very unsafe environment - e.g. parts of current-day Ukraine, or if they live together with someone abusive - where they are actually in constant danger. For them, it may make sense to feel unsafe all the time. (If you are one of them, I genuinely hope things get better for you soon.) But these are clearly situations where something has gone badly wrong; the feeling that one has in those situations shouldn’t be something that one was actively striving for. I think that any reader who doesn’t live in an actively horrendous situation would do better to feel safe most of the time. (Short timelines don't count as a horrendous situation, for reasons that I'll get into.)

As I interpret it, the core logic in both of the “you shouldn’t ever feel safe” claims goes as follows:

1. To feel safe implies a belief that nothing bad is going to happen to you
2. But something bad can happen to you at any time, even when you don’t expect it. In the case of AI, we even have reasons to put a significant probability on this in fact happening soon.
3. Thus, feeling safe requires having an incorrect belief, and the rational course of action is to not feel safe.

One thing that you might notice from looking at this argument is that one could easily construct an exactly opposite one as well.

1. To feel unsafe implies a belief that things aren’t going to go well for you.
2. But things can go well for you, even when you don’t expect it. In the case of AI, we even have reasons to put a significant probability on things going well.
3. Thus, feeling unsafe requires having an incorrect belief, and the rational course of action is to feel safe.

That probably looks obviously fallacious - just because things can go well, doesn’t mean that it would be warranted to always feel safe. But why then would it be warranted to feel unsafe in the case where things just can go badly?

To help clarify our thinking, let's take a moment to look at how the US military orients to the question of being safe or not. More specifically, to the question of whether a given military unit is reasonably safe or whether it should prepare for an imminent battle.

Readiness Condition levels are a series of standardized levels that a unit’s commander uses to adjust the unit’s readiness to move and fight. Here’s an abridged summary of them:

• REDCON-1. Full alert; unit ready to move and fight. The unit’s equipment and NBC alarms are stowed, soldiers at observation posts are pulled in. All personnel are alert and mounted on vehicles. Weapons are manned, engines are started, company team is ready to move immediately.
• REDCON-2. Full alert; unit ready to fight. Equipment except for NBC alarms is stowed. Precombat checks are complete, all personnel is alert and mounted in vehicles. Weapons are manned but engines are not started. Company team is ready to move within 15 minutes of notification.
• REDCON-3. Reduced alert. Fifty percent of the unit executes work and rest plans. Company team is ready to move within 30 minutes of notification.
• REDCON-4. Minimum alert. Observation posts are manned; one soldier per platoon designated to monitor radio and man turret weapons. Company team is ready to move within one hour of notification.

Now, why are all units not always kept on REDCON-1, or at least on REDCON-2? After all, there could always be an unexpected need for the units to mobilize or fight on immediate notice. Even units based on the US mainland might be called in to deal with a terrorist attack (as happened on 9/11) or natural disaster at any time.

The obvious answer is that a higher REDCON burns resources and makes the unit incapable of doing the tasks that it would carry out at lower readiness levels. The soldiers get tired, running the engines consumes fuel, soldiers who would be at observation posts aren’t carrying out observations, and work and rest tasks aren’t being carried out. 

Even though the military realizes that the unit could be needed at any time, setting their readiness condition isn’t just a question of whether it’s possible for the unit to be needed on short notice. It’s also a question of whether that’s likely enough to make the cost of maintaining a high state of readiness worth it in expectation.

I think that it makes sense for a individual person to also orient to the question of being safe or unsafe in a similar way. If someone claims that you shouldn’t ever feel safe, they are presumably saying that because they expect the feeling to translate to actions. They are saying that you should act as if you weren’t safe. But there is an opportunity cost to that; frequently thinking about possible threats burns mental cycles that could be used on something else, it makes it harder to rest and relax, and it biases the kind of information that you pay attention to.

In fact, I’m going to take the analogy a step further. I think that a person’s sense of (un)safety is in fact their subjective experience of an internal variable that tracks something analogous to the readiness condition level of that person’s body and brain. 

This quote from Cosmides & Tooby (2000) describes some of the effects they suggest may be triggered when a person is alone at night and feels a fear of being stalked; or in this analogy, the kinds of responses that the body and brain associate when being at their equivalent of REDCON 1 or 2:

(1) There are shifts in perception and attention: You may suddenly hear with far greater clarity sounds that bear on the hypothesis that you are being stalked, but that ordinarily you would not perceive or attend to, such as creaks or rustling. Are the creaks footsteps? Is the rustling caused by something moving stealthily through the bushes? Signal detection thresholds shift: Less evidence is required before you respond as if there were a threat, and more true positives will be perceived at the cost of a higher rate of false alarms. 

(2) Goals and motivational weightings change: Safety becomes a far higher priority. Other goals and the computational systems that subserve them are deactivated: You are no longer hungry; you cease to think about how to charm a potential mate; practicing a new skill no longer seems rewarding. Your planning focus narrows to the present: worries about yesterday and tomorrow temporarily vanish. Hunger, thirst, and pain are suppressed. 

(3) Information-gathering programs are redirected: Where is my baby? Where are others who can protect me? Is there somewhere I can go where I can see and hear what is going on better? 

(4) Conceptual frames shift, with the automatic imposition of categories such as "dangerous" or "safe". Walking a familiar and usually comfortable route may now be mentally tagged as "dangerous". Odd places that you normally would not occupy - a hallway closet, the branches of a tree - suddenly may become salient as instances of the category "safe" or "hiding place". 

(5) Memory processes are directed to new retrieval tasks: Where was that tree I climbed before? Did my adversary and his friend look at me furtively the last time I saw them? 

(6) Communication processes change: Depending on the circumstances, decision rules might cause you to emit an alarm cry, or be paralyzed and unable to speak. Your face may automatically assume a species-typical fear expression. 

(7) Specialized inference systems are activated: Information about a lion's trajectory or eye direction might be fed into systems for inferring whether the lion saw you. If the inference is yes, then a program automatically infers that the lion knows where you are; if no, then the lion does not know where you are (the "seeing-isknowing" circuit identified by Baron-Cohen 1995, and inactive in autistics). This variable may automatically govern whether you freeze in terror or bolt. Are there cues in the lion's behavior that indicate whether it has eaten recently, and so is unlikely to be predatory in the near future? (Savanna ungulates, such as zebras and wildebeests, commonly make this kind of judgment; Marks, 1987). 

(8) Specialized learning systems are activated, as the large literature on fear conditioning indicates (e.g., LeDoux, 1995; Mineka & Cook, 1993; Pitman & Orr, 1995). If the threat is real, and the ambush occurs, the victim may experience an amygdala-mediated recalibration (as in post-traumatic stress disorder) that can last for the remainder of his or her life (Pitman & Orr, 1995). 

(9) Physiology changes: Gastric mucosa turn white as blood leaves the digestive tract (another concomitant of motivational priorities changing from feeding to safety); adrenalin spikes; heart rate may go up or down (depending on whether the situation calls for flight or immobility), blood rushes to the periphery, and so on (Cannon, 1929; Tomaka, Blascovich, Kibler, & Ernst, 1997); instructions to the musculature (face, and elsewhere) are sent (Ekman, 1982). Indeed, the nature of the physiological response can depend in detailed ways on the nature of the threat and the best response option (Marks, 1987). 

(10) Behavioral decision rules are activated: Depending on the nature of the potential threat, different courses of action will be potentiated: hiding, flight, self-defense, or even tonic immobility (the latter is a common response to actual attacks, both in other animals and in humans). Some of these responses may be experienced as automatic or involuntary.

From this list, it’s pretty clear that it would be a bad idea to maintain this state all the time. Even if there were circumstances where it was theoretically possible for a person to be stalked, maintaining a constant state of fear would prevent them from digesting their food, relaxing, or for that matter thinking about anything other than how to get to safety. 

And if everywhere was classified as an unsafe state (as one does if they say you should never feel safe), then the priority of “get somewhere safe” couldn’t do anything useful. The person would just be stuck in a constant anxiety loop that necessitated constant running or fighting but never recognized a state where those responses could be even temporarily wounded down.

In a more recent paper, Levy & Shiller (2020) discuss our subjective experience of threat as being linked to a series of unconscious computations about the expected distance to physical danger, such as a predator. They describe a series of threat levels that one could see as being analogous to the readiness conditions of a military unit (extra paragraph breaks and bolding of the different stages added):

Encountering a life-threatening situation engages neural computations that consider information about the environment and the source or threat. These computations design defensive policies and select adaptive responses for execution. The threat imminence continuum model [2], which maps defensive behaviors onto levels of threat imminence (how far a predator is in time and space), provides a platform of prey-predator relations for assessing the neural circuits and computations for survival [3, 4]. 

In the first ‘safe’ stage, there is no threat, an encounter with a predator may occur in the distant future. Individuals may experience occasional anxiety, and flash forward toward possible future threats. Cognitive control and emotion regulation could keep this process in check. 

Next is the ‘pre-encounter threat’– the predator is not present, but may surface at any moment. Individuals may experience anticipatory anxiety and exhibit vigilance and preparatory behaviors. 

In the more dangerous ‘post-encounter threat’ the prey, not yet detected, observes the predator. This step generates encounter anxiety, involving close inspection and anticipation of the predator’s moves, strategic freezing to avoid detection and gather information, and avoidance estimation. 

Finally, the prey is under most extreme danger during the ‘circa-strike’ phase, when the predator is attacking. Within that attack mode, the predator could be distant enough to allow a feeling of fear and rapid thoughts examining the situation and assessing escape routes. Fight or flight ensues as the predator gets closer yet without contact.  The final point of contact provokes hard-wired, fast, often poorly-executed reactions of freezing and panic [5].

If this model is right, it then implies that “feeling safe” does not imply an assessment of zero probability of threat. Feeling safe is the subjective experience of the brain assigning a low probability to a threat, and emphasizing the kinds of behavioral and biological priorities that make the best use of a low-threat situation. 

Meanwhile, “feeling unsafe” implies an assessed level of at least ‘pre-encounter threat’, meaning at least a moderate probability of an immediate physical threat. “Moderate” in this context is a bit of a fuzzy term, given that even something like a 1% probability of there being a predator around could plausibly make it justified to maintain this level of readiness.

Still, a 99+% probability of being safe isn’t that high of a bar; extreme probabilities are common. If a person’s risk of being physically attacked was 1% per day, they would have a 97% chance of being attacked within a year. Some of the people reading this post may live in a location where their risk of being assaulted is around that order of magnitude, but if they aren’t, then constantly feeling unsafe implies that their subconscious isn’t calculating the probabilities correctly. (Probably warranting a diagnosis of an anxiety disorder.)

There is also the consideration that feeling unsafe doesn’t only imply a moderate probability of something bad happening; it implies a classification of the bad thing as “the kind of a thing that this type of a readiness response is useful for dealing with”. To extend the analogy to the military’s readiness levels, having your weapons manned and being ready to fight isn’t very useful if the threat being faced is an infectious disease, or Congress deliberating a funding cut that involves the unit in question being decommissioned. 

Likewise, it isn’t very useful to activate behavioral responses that are evolved for the purpose of getting to safety from an imminent threat, if the threat involves unaligned AI possibly being developed within the next decade or so. There isn’t anywhere that you could run away to, nor a concrete enemy you could defeat, so this response would be stuck trying to do an impossible task. The mindset necessary for solving the problem is clear and calm thinking, and an ability to relax and get rest as well. So this is an instance of a situation where it might be warranted to feel safe even if you intellectually acknowledge that you might not be safe.

So to recap, I think that “you should never feel safe” is an incorrect argument for several reasons:

1. It implies at least a moderate probability of an immediate threat
2. It assumes that the kinds of threats the person is facing, are ones that are effectively dealt with by keeping the mind and body in a constant state of low-grade anxiety
3. It assumes that it’s possible for those anxiety responses to achieve something useful even when every possible state you can be in is classified as unsafe, and their very purpose is to get you somewhere safe

If you are somewhere where there is an actual tangible threat against you - then yes, feel unsafe! If you are approached by someone you know to be violent or abusive, or if you are out on a walk and you think that you might actually be stalked - then yes, feeling unsafe may very well be the right response. 

But if those criteria aren’t met, you are probably better off feeling safe, and harnessing the resources that that state grants you.

1. ^{^}

   Malcolm Ocean describes a form of this experience:

   I recount how in 2019, I heard a podcast where Esther Perel says to a client about his partner who has PTSD flashbacks, “you can tell him ‘you’re safe now’” and I found myself thinking “that’s not okay. I can’t feel that I’m safe in this moment. AI could eat the world, and I’m not doing enough about it. I can’t feel safe until we’ve figured it out.”