A couple of chess observations related to epistemology and alignment.

Queen Odds

Take the starting chess position. Remove the black queen.

rnb1kbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1

https://lichess.org/analysis/rnb1kbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR_w_KQkq_-_0_1

It's overwhelmingly likely that White is winning.

But just how sure should we be of that, epistemically? We don't have a mathematical proof yet.

Even more absurd question: how sure should we be that Black is not winning? There's no mathematical proof of that either.

I claim that every educated chess player should believe that White is winning, (almost) as certainly as if they had a mathematical proof. Like, I mean that it should be much more certain than mathematicians' consensus on P vs. NP or Goldbach's conjecture or the twin prime conjecture etc.

Reasons to believe:

1. We know all types of pawn structures which may arise in the opening/middlegame. We know there's no way to force a fortress (= lock the pawns/position) if you're down an entire queen.
2. We know all methods of creating tactical possibilities in normal positions. We know there's no way to force tactics if you're down a queen OR win back enough material with those tactics.
3. If Black could create enough problems for White to win back an entire queen, White could use the same method to crush Black. While the formal strategy stealing argument can't be applied to chess because of zugzwang and because the position isn't 100% symmetrical (Black doesn't have a queen), informally we know enough about chess to know that Black won't force a zugwang or won't benefit enough from the empty D8 square.
4. We know enough about types of chess positions to know there's no way to avoid trades while being down a queen.
5. We know enough universal chess plans (such as doing a pawn storm against a fianchetto) to know they are unstoppable if you're down a queen.

Basically, I'm saying our knowledge of chess positions is "exhaustive" on the level of approximation of being up/down an entire queen. There's not enough place to squeeze enough uncertainty that Black could survive, let alone win.

Conclusion: there's an intersection between statistical reasoning and logic; informal reasoning has structure which can be studied.

Material Advantage

Consider the following heuristic:

"Material advantage often leads to winning a game (X)"

Now, consider a combination of heuristics:

"Material advantage often leads to winning a game (X), because it often allows to win more material by overwhelming the defences (Y) and certain amount of material advantage guarantees a win under normal conditions (Z)"

Y and Z are additional heuristics (mixed with logic) which help to evaluate how much X is true in a specific situation. I.e. if you notice that your material advantage doesn't help to win more material, it's a sign that X is not true in your situation (probably because the opponent has built a fortress).

Conclusion: informal reasoning has structure which might look like "try to fill the gaps between a heuristic (X) and certain knowledge with auxiliary heuristics (which describe assumptions about the way X works)".

Though that requires your goal to be well-defined (like "give checkmate") and heuristics defined over a shared ontology and knowing how to find good auxiliary heuristics.