Previously:

My parents taught me the norm of keeping my promises.

My vague societal culture taught me a norm of automatically treat certain types of information as private.

My vague rationalist culture taught me norms that include:

noticing when I'm confused
noticing when societal norms were inadequate to handle my situation
being very honest, and having strong theoretical underpinnings for handling situations where I felt it was inadvisable to be honest.

Eliezer's post about meta-honesty was one of the most influential posts I've read in the past few years, and among the posts that inspired the coordination frontier. I was impressed that Eliezer looked at ethical edgecases, and wasn't content to make a reasonable judgment call and declare himself done.

He went on to think through the ramifications of various policies, devise a potential new norm/protocol, examine reasons that protocol might work or not work. He noted considerations like [paraphrased] "It matters that the norm be simple enough that people can reliably understand and use it." Or, quoted directly: "This norm is too subtle for Twitter. It might be too subtle for us, too."

From this post, I derived a (not-quite-spelled-out) norm of "when you try to navigate the edge cases of your norms, try thinking through the underlying principles. But don't try to be too clever, and consider the ways your edge-case-handling may fail to scale."

With this in mind, I want to walk through one of the ethical dilemmas I faced that I reflected on, when writing Norm Innovation and Theory of Mind. This is more of an object-level post, primarily a followup to my Privacy Practices sequence. But it seemed like a useful illustrative example for the Coordination Frontier concept.

Privacy norms can be wielded as an obfuscating weapon

Sometimes, privacy is wielded as a tool to enable manipulation.

I’ve run into a couple people who exploited my good faith / willingness to keep things confidential, as part of an overall manipulative pattern. Unfortunately, I don't feel comfortable going too far into the details here (please don't speculate in the comments), which makes it a bit harder to sanity check.

"Manipulation" is a tricky to define. It's a blurry line between "communicating normally" and "communicating in a way th...