Note: This post was formatted and worded with the help of an AI assistant. The ideas, thoughts, and reasoning are entirely human—mine. Not everyone’s a polished writer, and sometimes getting ideas out clearly needs a little help. That’s all this is.
When I first started using large language models (LLMs), I honestly saw them as glorified autocomplete machines—clever, predictive, and sure, sometimes useful. But the more I explored them, the more I started noticing possibilities beyond surface-level word prediction. Especially now, with the rise of agentic systems and protocols like MCP (Model Context Protocol), LLMs are starting to behave more like tools with agency—calling APIs, triggering workflows, even making decisions in context.
This shift is exciting, but also fragile. With access to external tools, things like prompt injection attacks become more than theoretical—suddenly they’re risks with real-world consequences. That might be one reason we haven’t seen these capabilities adopted at massive scale just yet.
Still, something caught my attention recently, and I wanted to get the thought down.
LLMs generate responses by sampling from a probability distribution over the next token. That sampling can be shaped using two key parameters:
Most models—especially closed ones—lock these to default values optimized for perceived coherence. But “coherent” doesn’t always mean accurate, and I don’t think there’s a one-size-fits-all value for either of these parameters.
So I started wondering—what if we didn’t treat those values as fixed?
Instead of answering with a single response at a fixed top-p and temperature, imagine this:
Is this compute-intensive? Absolutely. But it might also help surface more reliable outputs in certain contexts—especially ones where hallucinations carry a higher cost.
This kind of multi-sample, score-and-select approach isn't feasible at scale right now. GPU time is costly, and energy constraints make things worse. But that bottleneck won’t last forever. With growing attention on nuclear fusion, renewable resources, and more efficient compute architectures, it’s possible we’ll see a shift in what’s affordable to run in real time.
If the cost of compute drops—or the value of high-confidence outputs rises—this method might start to look less ridiculous and more like a viable direction.
Is this idea already out there? Maybe. It feels obvious enough that someone’s likely thought about it or even implemented a variant of it in internal eval systems. But that doesn’t really matter here. What matters is the reasoning path that led to this thought—and the realization that accuracy isn’t just about improving models, but also how we sample from them.
Postscript
This isn’t a proposal, or a whitepaper, or even a serious recommendation. It’s just a piece of mental scratchwork. A reminder that sometimes even small changes—like treating parameters as levers instead of constants—can open up whole new ways of thinking.