This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Large language models today have a basic problem: they tend to treat information as equally valid if it appears often enough in their training data. They don’t do a good job distinguishing high-quality evidence from low-quality or biased sources. This leads to several recurring issues:
They confidently state things that are simply wrong.
They give smooth-sounding answers even when they’re uncertain.
They don’t reliably remember or apply corrections from previous conversations.
They struggle with deep, step-by-step reasoning over time.
I think we can do better. What if we gave the model a permanent, verified internal knowledge base that acts as its primary source of truth, and then built a mechanism to gradually improve that base through rigorous testing?
Here’s the basic design I have in mind:
The model would start every answer by checking this internal knowledge base first. High-confidence facts in the base would anchor its reasoning. The base would also be made public (something like an auditable wiki) so others can scrutinize and contribute to it.
Each fact in the base would have a confidence score, full history of how it got there, and where the information came from. Certain bedrock truths — things like the laws of classical physics or basic principles of evidence — would get special “constitutional” status. These would be very hard to override.
The key innovation is what I’m calling a ratchet mechanism. When new information comes in, the system doesn’t just replace old beliefs. It forces competing claims to compete against each other and against the existing high-confidence knowledge. A claim only loses credibility if it’s actually undermined by strong evidence. High-confidence conclusions get extra protection, but never absolute immunity.
The system would keep multiple competing explanations alive when the evidence is genuinely mixed (I call this “equipoise”). It would clearly signal uncertainty instead of pretending to be sure. Over time, the best-supported positions would gradually gain confidence while weaker ones fade — but nothing gets erased. The full history stays visible.
I also think the ratchet could eventually run internally, with different parts of the model arguing against each other before settling on the strongest answer.
Why This Matters Right now, even the best models are still heavily influenced by whatever was popular or convenient in their training data. In fields like medicine and drug development, where I work, getting things wrong can have real consequences for patients. A system that can steadily improve its reliability by testing claims against solid anchors and accumulated evidence would be genuinely useful.
Consider a drug being tested in two different cancers. Early data in one cancer looks terrible. Later lab results suggest the drug might behave differently in the second cancer due to a biological distinction. The system would keep both possibilities alive, adjust confidence as better data arrives, and never pretend the early failure didn’t happen.
I’m not claiming this solves everything. There are real engineering challenges around speed, storage, and maintenance. But I believe this general direction — building a ratcheted, auditable internal knowledge base with truth-seeking as the top priority — is worth exploring.
Author Note I’m a physician and lawyer working on new therapies for brain tumors. I’ve spent a lot of time testing frontier AI models because I want tools I can actually trust when lives are on the line. Frequent obvious flaws in how current models analyze clinical data are hard to accept.
Would anyone be interested in discussing or developing this further? What problems or improvements do you see?
Large language models today have a basic problem: they tend to treat information as equally valid if it appears often enough in their training data. They don’t do a good job distinguishing high-quality evidence from low-quality or biased sources. This leads to several recurring issues:
I think we can do better. What if we gave the model a permanent, verified internal knowledge base that acts as its primary source of truth, and then built a mechanism to gradually improve that base through rigorous testing?
Here’s the basic design I have in mind:
The model would start every answer by checking this internal knowledge base first. High-confidence facts in the base would anchor its reasoning. The base would also be made public (something like an auditable wiki) so others can scrutinize and contribute to it.
Each fact in the base would have a confidence score, full history of how it got there, and where the information came from. Certain bedrock truths — things like the laws of classical physics or basic principles of evidence — would get special “constitutional” status. These would be very hard to override.
The key innovation is what I’m calling a ratchet mechanism. When new information comes in, the system doesn’t just replace old beliefs. It forces competing claims to compete against each other and against the existing high-confidence knowledge. A claim only loses credibility if it’s actually undermined by strong evidence. High-confidence conclusions get extra protection, but never absolute immunity.
The system would keep multiple competing explanations alive when the evidence is genuinely mixed (I call this “equipoise”). It would clearly signal uncertainty instead of pretending to be sure. Over time, the best-supported positions would gradually gain confidence while weaker ones fade — but nothing gets erased. The full history stays visible.
I also think the ratchet could eventually run internally, with different parts of the model arguing against each other before settling on the strongest answer.
Why This Matters
Right now, even the best models are still heavily influenced by whatever was popular or convenient in their training data. In fields like medicine and drug development, where I work, getting things wrong can have real consequences for patients. A system that can steadily improve its reliability by testing claims against solid anchors and accumulated evidence would be genuinely useful.
Consider a drug being tested in two different cancers. Early data in one cancer looks terrible. Later lab results suggest the drug might behave differently in the second cancer due to a biological distinction. The system would keep both possibilities alive, adjust confidence as better data arrives, and never pretend the early failure didn’t happen.
I’m not claiming this solves everything. There are real engineering challenges around speed, storage, and maintenance. But I believe this general direction — building a ratcheted, auditable internal knowledge base with truth-seeking as the top priority — is worth exploring.
Author Note
I’m a physician and lawyer working on new therapies for brain tumors. I’ve spent a lot of time testing frontier AI models because I want tools I can actually trust when lives are on the line. Frequent obvious flaws in how current models analyze clinical data are hard to accept.
Would anyone be interested in discussing or developing this further? What problems or improvements do you see?