Failure mode is a term for a, usually common, way things fail when attempting something. for example, confirmation bias is a common failure mode in reasoning.
Knowing and understanding possible failure modes in what you attempting to do is important in order to avoid them. Security Mindset and Ordinary Paranoia discusses the difference between finding and fixing failure modes by trying your best to imagine all the ways your system could fail ("ordinary paranoia") vs having a tight argument that your system does not fail (under a small number of assumptions which are each individually quite probable).
See also: Postmortems & Retrospectives
Status Quo Bias
Heuristics & Biases
Mind Projection Fallacy
Pitfalls of Rationality