Have you looked at a Cohen's kappa based system. Treat R1 and R2 as raters scoring rollouts. Weighted kappa lets you assign higher cost to disagreements on rollouts that matter more (high-stakes, high-update, etc.), which addresses your point about some rollouts being more critical for learning. And Fleiss's kappa generalizes to N reward functions if you want to measure agreement across an ensemble rather than just a pair.

The open question is whether κ(R_true, R_proxy) bounds policy regret in the way that EPIC/STARC metrics do for continuous rewards. My intuition is yes for discrete/categorical reward signals (preference judgments, pass/fail), where the L2-family metrics those papers use aren't natural anyway. Happy to discuss, working on empirical substrate for this.