Hey all — I just submitted my Master’s thesis, and would really appreciate feedback from people thinking about non-task value alignment or LLM transparency metrics.

I explored whether Reinforcement Learning from Human Feedback (RLHF), using a constitutional rubric, could align a language model with transparency as a civic value. The case: Amsterdam’s parking permit system.

Methods:

Model: Mistral-7B-Instruct v3
Training: PPO using a 6-dimensional transparency rubric (Explainability, Completeness, Source Attribution, Procedural Clarity, Accuracy, Limitation Disclosure)
Baseline: SFT using ~300 examples
Evaluator: Independent LLM Judge (Claude Sonnet 4) scoring per-dimension across 300 held-out citizen queries

Key Results

RLHF win rate: 61% vs SFT baseline (p < 0.001)
Biggest gains:
- +21.6% Information Completeness
- +19.5% Source Attribution
Accuracy dropped by -6.6%, mostly due to over-hedging (e.g. "This may be different for your city", despite giving city specific information)
81% of reward signal came from explicit constitutional constraints, not emergent learning (according to the Optuna trials)

Takeaways

RLHF can encode democratic values — but needs deliberate constraints
There’s a fundamental tradeoff: transparency ≠ precision
Technical alignment isn’t enough — civic values require governance, citizen input, and deliberation

Would love feedback — especially on:

whether this rubric-based evaluator is generalizable
how others might have measured non-task values like “transparency”
if this work might overlap with existing alignment metric toolkits

I’ll post the paper & code when cleaned up.

Abstract

Large Language Models in government services often lack alignment with democratic values, particularly transparency, risking opaque AI-citizen interactions that undermine public trust. This thesis investigates whether Reinforcement Learning from Human Feedback (RLHF) can systematically embed transparency in government AI systems.

Using Amsterdam's parking regulations as a critical case, we fine-tuned a Mistral-7B model through constitutional RLHF, where a six-dimensional transparency rubric guided Proximal Policy Optimization. This approach was benchmarked against supervised fine-tuning across 300 citizen queries, with an independent LLM evaluating transparency outcomes.

The RLHF model achieved a 61%-win rate (p < .001), with substantial improvements in Information C...

Posts

Wikitag Contributions

Comments