There are a lot of senses in which I or the people around me can be considered unsafe. Many-tonned hunks of metal whiz by us on the same streets we have to navigate on foot to buy our groceries. The social infrastructure by which we have access to clean drinking water is gradually being adulterated. Our country is run by occasionally genocidal white nationalists. And, of course, The Bomb. But when I hear people talk about feeling unsafe, they are almost never describing a concrete threat to their physical well-being. (As usual, life may be different for the less privileged classes, who have reason to fear the authorities, and behave accordingly.) "Safety" does not come up as a motive for actions taken or avoided in order to mitigate such threats. Instead, it seems that "safety" nearly always means a nonjudgmental context (the exact opposite of what I would naively expect to be able to ensure clean drinking water or keep the cars from colliding with us), and "feeling unsafe" is generally used to explain only why they're trying to withhold information (mainly "vulnerable," i.e. relevant-to-their-interests, information) in a way that seems out of proportion to actually existing risks and opportunities.
Judgment and Punishment
Consider a simple model: information about social censure consists of two parts. Each socially legible action is assigned a vulnerability score based on how often, empirically, someone responds by revealing the intent to punish the actor. Actions are sometimes defined contextually, so that talking loudly in a crowded bar or on the street is different than talking loudly in a library or theater, but it's not a different action depending on who is present - only impersonal context cues and stereotyped identities (e.g. some things are inappropriate "in mixed company"). Vulnerability is a global variable with respect to persons.
If Cato is observed to punish singing but not dancing, and Flaccus is observed to punish dancing but not singing, this is treated as unpredictable random variation - possibly just measurement error. Cato and Flaccus both acquire a reputation for judginess, and both singing and dancing start to feel like vulnerable activities, so people will feel inhibited about doing either activity in the presence of either censor.
At the same time, each known person is evaluated for their generic propensity to punish, or judginess. Some people will physically attack you for violating norms (they often wear dark blue or gray-green), others will just yell at you, still others will politely hint that others might disapprove, and a few are universal receivers, totally nonjudgmental. Revealing others' intent to punish is considered a veiled threat, and is therefore itself a mild form of intent-to-punish. To be nonjudgmental, one must deny others information about what is likely to be punished elsewhere.
We recognize judgmental people not merely by their actual punishment behavior (in the ancestral environment, where ostracism could easily be permanent and deadly, this might have been playing things quite a bit too close), but by their posture, the patterns of tension in their voice, and so on.
I think that this model fits how people in our society experience a sense of social safety or unsafety surprisingly well for something so simple. One virtue of this model is that it correctly predicts that "code-switching," i.e. adjusting to variations in standards between different cultures for the same activity in the same context, is more difficult than learning different behaviors for different contexts within a single culture. Code-switching imposes a greater cognitive load due to its strong dependence on theory of mind.
The Information-Suppression Field
One important characteristic of this setup is that it structurally advantages information-suppression tactics over clarity-creation tactics.
If I try to judge people adversely for giving me misleading information, I end up complaining a lot, and quickly acquire a reputation for being judgmental and therefore unsafe. Ironically, I get more of the behavior I punish, since being categorized as judgy leads to people avoiding all vulnerable behaviors around me, not just the ones I specifically punished. I cut myself off from a lot of very important information, and in exchange, maybe slightly improve the average punishment function - but this would provide an information subsidy to all other judgy agents, even ones whose interests conflict with mine and are trying to prevent me from learning some things. And most likely I just add to the morass of learned inhibitions.
On the other hand, if I wish to suppress some information - say, that some enterprise I'm profiting from is fraudulent - and I don't otherwise read as unsafe, then I can very slightly punish it - say, by gently discouraging people from talking about it because it seems likely to be harmful, because it hurts some people's feelings, etc, If I only need to suppress a few pieces of information, and there are other REALLY judgy people out there, then I can externalize most of the enforcement costs onto either the actual judgy people or the imaginations of the people I am manipulating.
A simple example:
Alice has a pervasive sense that she is being cheated in life somehow, and lashes out from time to time at people who seem like they're piling on. Carol has a consistently gentle, positive vibe, and owns a drugstore from which Alice regularly purchases expensive homeopathic medicines. Bob, who knows both of them, starts to tell Carol about how he's done some thinking about it, and homeopathy seems to him like it couldn't possibly work. Carol hints to Bob that this is a sensitive subject. Bob reasons, implicitly, that if even Carol doesn't like him talking about his idea, he had darn well better make sure not to talk about it around Alice.
This is an adversarial game that different secretive coalitions can play against each other, at the expense of other people trying to use censure for other reasons. All such moves, however, also benefit nonjudgmental people, who can collect a surplus from living in a society that relies on standards, while collecting a disproportionate amount of information and social capital by never contributing to attempts to track and censure misbehavior.