There are many conflicting opinions about how useful AI is for coding. Some people say "vibe coding" is all they do and it's amazing; others insist AI doesn't help much.
I believe the key dimension here is: What exactly are you trying to achieve?
(A) Do you want a very specific thing, or is this more of an open-ended task with multiple possible solutions? (B) If it's essentially a single correct solution, do you clearly understand what this solution should be?
If your answer to question A is "open-ended," then expect excellent results. The most impressive examples I've seen typically fall into this category—tasks like "implement a simple game where I can do X and Y." Usually, open-ended tasks tend to be relatively straightforward. You probably wouldn't give an open-ended task like "build a better AWS."
If your answer to question A is "a specific thing," and your answer to B is "yes, I'm very clear on what I want," then just explain it thoroughly, and you're likely to get satisfying results. Impressive examples like "rewrite this large complex thing that particular way" fall into this category.
However, if you know you want something quite specific, but you haven't yet figured out exactly what that thing should be—and you have a lot of coding experience—you'll probably have a tough time. This is because "experienced coder doesn't know exactly what they want" combined with "they know they want something specific" usually means they're looking for something unusual. And if they're looking for something unusual but struggle to clearly communicate it (which they inevitably will, because they're uncertain), they'll constantly find themselves fighting the model. The model will keep filling in gaps with mundane or standard solutions, which is exactly what they don't want.
(There's ofc also (C) How obscure is the technology? but that's obvious)
Well, today GPT-5-Codex solved it on the 2nd try. (The first version it gave was already conceptually correct, but I guess had some subtle bug. After I told it to fix it and test the fix, it gave a working solution.)
I am just surprised how well the agentic loop is working. It cloned the specific Lean version's source code I was asking for, inspected it to understand the data structure, downloaded a release tarball to test it, all without losing track of its goals. All this would have been unimaginable ~a year ago.
So yeah, in 7 months (but maybe even 2 if you count the base GPT-5 attempt) we went from "not even close" to "solved" on this problem. Not sure how I should feel about this...