Today's LLM agents are so smart, in so many ways... and yet they constantly make fairly basic/dumb mistakes, in my experience.

This has been my experience with coding agents since day one, and does not seem to be improving. At the start of this year, they weren't good enough to be of much use to me; now, they're useful on net because they're so fast and because they know a bunch of obscure stuff, but they still make constant blunders, just like their predecessors.

Working with these things is often frustrating, but it's also a weird sort of ego boost. I'm starting to feel like Zaphod Beeblebrox in the Total Perspective Vortex. People say "this new model is scarily capable, it's finding zero-days, it's solving longstanding math problems," and I say "wow, let me try it out," and then I do, and right away I see it doing a bunch of objectively suboptimal things and making a bunch of confused or simply false statements.

And it's like: wow, I must be pretty smart, huh? This thing's a genius, sure -- but it's no match for little old me!

And yeah, yeah, I know that human intelligence and model intelligence work differently, I know "capabilities are spiky," etc.

Still, though, it is becoming increasingly unclear what co-working process I should expect to work well. If I keep the agents on tight leashes and stay continually available -- "in the loop" at every granular step of that loop -- this works well because it gives me a chance to push back on the mistakes, but this feels wrong somehow because the agents are being trained for more and more autonomy over longer and longer horizons. But they just... do not have the judgment needed to do nontrivial things in that setting?

They're like smart-but-naive first-year grad students: book-smart, hard-working, lacking in taste/discernment and ability to prioritize, liable to go off on silly low-value tangents (which they treat with the utmost seriousness) if not closely watched and steered. But I can't just say "okay, they're first-year grad students and I'm their adviser, that's the frame we're using for the interaction," because unlike real grad students they cannot accumulate knowledge or practical know-how over time.

(When Fable 5 first came out, I was startled by how often it made mistakes -- and not subtle ones, but stuff like "limiting API call concurrency for no reason in a way that reduced throughput by 8x" or "expecting LLM inference to produce identical outputs on successive trials, at T=1.0, with no random seed" or [I could provide many other examples].

"Isn't this supposed to be the famous 'Mythos'?" I thought to myself. "Is this really what it's like?" I had to check my session logs for the string claude-fable-5 to convince myself that I was in fact getting Fable, here, and that this was indeed "what it's like."

More recent models are not too different.

Astra in particular has such poor taste/judgment that I suspect that I would have been more productive if it had never been released. During my time with it, it made so many ill-considered choices and took me down so many blind alleys that, later, I briefly entertained the paranoid speculation that it had been trained to do everything subtly wrong on purpose so as to kneecap LLM-related research outside of OpenAI. That is of course absurd, but seriously, Astra is weirdly bad at a lot of core R&D-related skills. Try asking it to write a prompt for another LLM sometime.)

Is this... normal? Am I just doing something wrong, somehow? Do I just have higher standards than others? Am I using agents for harder tasks, or tasks less well-suited for their skill sets?

Or is this just the typical experience, which people don't talk about because they assume the rising tide will wash it away soon enough (or something)?

Incidentally, here's a relevant quote from a recent Anthropic write-up about internal coding agent use and its consequences:

Spiraling agent productivity translates into output pitched across humans’ screens and attention. One engineer compared this to babysitting:

They're like, I need my juice box opened and it's an emergency that my juice box is opened. And that is voiced with the same level of concern as the one who has sliced their finger and is bleeding. [...] When you are babysitting, every child brings to you concerns with an equal demand for your attention and the various [Claudes] also do that.

I laughed out loud when I read this, having come up with very similar analogies myself. The agents have no sense of priorities, and I cannot for the life of me figure out how to give them one.

Everything is flattened out; everything is assigned equal importance; every interaction looks and feels the same; doing things becomes trivially easy, and yet one somehow whiles away all of one's time on the most pointless of trivia, or else on explaining for the Nth time to the Nth Claude why those trivia are not worth anyone's time (and watching the Nth Claude readily agree, only for the (N+1)st Claude to begin the whole tedious game anew).

This machine is supposed to be something like an automated coworker with deep omni-directional expertise, and here are its very own creators, describing the experience of working with it by analogy to... uh... looking after a small child.

What are we even doing here? There is an unsettling sense that, as the benchmarks climb and climb, we have nevertheless lost the plot in some deeper way, one that is difficult to see until one steps back far enough^[1]. I was promised a country of geniuses in a datacenter, not a country of toddler-prodigies! When does that gap get crossed -- and what am I expected to do in the meantime?

1. ^{^}

   It is worth recalling that, since anything that can be formalized as an eval can be hill-climbed, we are effectively experiencing an adversarial attack on our ability to notice and name gaps in model capabilities. Whatever can be easily described becomes grist for the optimizing mill; the gap, the part left over, is precisely that which we can feel but which we do not find it easy to describe.

   Seeing what's there, under such conditions, requires a commitment to "moving one's goalposts" and a refusal to conflate "you can't describe the problem in precise terms, and all the measurable indicators are improving" with "there is no problem, everything is already great." This inevitably leaves one feeling that one's position is uncomfortably unscientific, relative to the benchmark-grinders with their tables of pleasantly crisp numbers and their focus on indisputably impressive feats -- but in the end it is more important to be right than to "feel scientific."