faul_sname's Shortform

faul_sname

This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Today, I needed to work through substantial project with a lot of drudgery (checking through an entire 1M+ LOC codebase for an http api for patterns which could cause state leakage between requests if we made a specific change to the request handling infrastructure. This involved a mix of things which are easy to do programmatically and things which require intelligent judgement, and has a fairly objective desired artifact (a list of all the places where state could leak, and a failing functional test demonstrating that leakage for each one).

I decided to do the John Henry thing - I set up Claude Code (in a container with --dangerously-skip-permissions) in one worktree with a detailed description of the project and the task, and then in a separate worktree I set off with my favorite text editor and without the help of any AI tooling more advanced than Copilot.

I finished about 4 hours later, despite fairly frequent interruptions to provide clarification and further instructions to Claude. Claude is now reaching the 7 hour / 100M token mark and has still not finished, though it has multiple times now declared that it has succeeded at the task and that the codebase is safe for this migration (it's not).

I'm honestly pretty shocked, because this task seemed like a pretty much perfect fit for a coding agent, and is one that doesn't require all that much codebase-specific context. I went into this expecting to lose - I was trying to quantify how much coding agents can help with obnoxious repetitive maintenance tasks, thus allowing maintenance work which might otherwise have been deferred to happen at lower cost. But I guess that's not the post I'm writing today (which is a bummer, I had a whole outline planned out and everything).

Likely this situation will change by next year, but for now I suppose the machine cannot yet replace even the more repetitive parts of my job. Perhaps things are different in ML land but I kind of doubt it.

the AI will often just repeat the code block in B and C so that it doesn't have to delete anything in A

Some human devs do this too. In the short term it reduces the likelihood of breaking things because something you weren't aware of relied on the old version. In the long term it makes changes harder, because now if you want to change the logic instead of changing it in one place you have to change it in n similar but usually not identical places, and if those places are different in ways that affect the implementation, now you have to try to make an in... (read more)

1Smaug12310mo

I would strongly recommend asking Claude to set up a deterministic system to do this. Depending on precisely how flexible you need to be, Semgrep might be the correct tool for the job, and Claude should be able to configure that. It may be able to suggest other deterministic approaches using tools I don't know about.

2faul_sname10mo

In my experience, semgrep does not play well with trying to find cross-class behavior in dynamically typed codebases with lots of dependency injection, which is why I was trying to make Claude write some code which combined static analysis (in the form of reflection or ast parsing) with runtime logic for gathering information which is hard to determine statically but easy to determine at runtime. For reference the code I ended up writing for this part was about 40 lines, it wasn't very complicated. Trying to do it in full generality purely by static analysis would be insanely complex (because php has terrible constructs like $$foo = "bar" and $foo->$bar = "baz", which this codebase doesn't use and can be trivially verified not to use, but which would be a nightmare to handle if they were used), but fortunately this wasn't what I needed. But yeah, I also expected Claude to be able to do this trivially. It is able to trivially do most tasks which feel, to me, to be about this difficult or even a bit more difficult. This task felt like it should have been easier, since it's one where there's a lot of available signal to self-correct if you make a mistake, much more so than for many of the "build and test a feature" style tasks that Claude regularly does with no drama. Which is why I thought it would be a good example for a post along the lines of "many people use LLMs to quickly add sloppy features to their codebase, increasing technical debt, but it's also possible to use them to resolve technical debt much faster than doing it by hand". And then I tried it.