Y'know, I did eventually get a smartphone. But even today, I use my phone almost exclusively for maps and private messaging/calls. I have zero social media apps on it, and view zero shortform video content on it. And I think that's basically the right attitude.
Drawing the analogy to LLMs: there are some narrow use-cases for which LLMs are great. Even two years ago, they were already great as a search engine alternative, including generating simple StackOverflow-style code snippets. In the last few months, they've also become viable for generating nontrivial numerical code and (sometimes, with a fair bit of scaffolding and Lean) nontrivial proofs. These use-cases are to LLMs what maps and direct messaging are to smartphones.
... but then there's also a whole slew of other LLM use-cases which are more analogous to social media or shortform video on smartphones. And those use-cases I still avoid like the plague. Much like both social media and shortform video, they seem addictive and actively mind-killing. As a rough heuristic, if it feels like the LLM is thinking with you or for you, then that's usually a use-case to avoid scrupulously.
(You might wonder if using LLMs for proofs feels like they're thinking with/for me, but mostly it feels to me like they're throwing prodigious amounts of shit at the wall and seeing what sticks, with pretty little thinking-of-the-sort-I-do at all. The power of Lean verification is that it makes that sort of brute-force-y search a viable method for finding proofs.)
1makes sense--- my razor for "this is like the evil social media notification slop version of LLMs" vs "i'm uplifting in a way that is good for me" is whether or not I can defend an artifact by myself on a whiteboard. I almost never meet this bar. So when I use it to teach me some new tool, then assemble that tool into a solution to a problem, I would ideally slow down enough to let my own alone-at-a-whiteboard command over the material catch up. Perhaps we agree that finding time to do this is the analogue of not letting social media notifications fry your brain.
(I think my work is less conceptually complicated or interesting than your work)
1I thought about adding commentary but decided the quote's recontextualization speaks for itself.
That shortform (which John may or may not have remembered writing) was living in my brain ever since I read it, as a poweruser who occasionally has pangs of "what if enfeeblement?" and so on. So to me, the post about how things shifted as of the last few months feels something shaped like "the one guy we could count on to resist enfeeblement is now uplifted" which feels like a "sign of the times" of some sort.
31I think neither one year ago nor now nor anytime soon will "LLMs improve productivity" have a clear binary answer. It's a jagged landscape that depends on things like
- The domain
- The concrete task
- The human's experience
- How they're using the LLM
- How conscientious they are in the process
- ...and 50 other things
And with every year, a larger part of this landscape comes up positive for LLMs.
Ironically, I can imagine that part of what leads to the relative improvement of LLM performance is that (some) humans, due to extended LLM use, have seen their skills atrophy, and hence the bar was lowered. I think it's very difficult to use such technology without "moving backward", as John has put it.
1either simple/well-represented in training
I believe most people are on board with the "simple" category with plus or minus 6 months of model capabilities about their version of "simple"... But I'd like to document an example about how jagged the border around "well-represented in training" can be:
Using Fable and Opus 4.8 on an open source project that already existed in 2025 for rendering chatbot transcripts with React JS (one of the most common in-distribution coding tasks[1]), it managed to "simplify" stuff by adding more and more conspiracy-theories-adjacent code from its own playwright testing (and those fixes somehow got approved by 5.5 or 5.6 Sol reviews ..but only when Fable asked itself for the reviews) even after I explicitly asked it something like "stop using the slop offset theatre, the correction on main branch is obviously in the WRONG direction [screenshots] and not related to the element from which it's dynamically calculated at all but to the header outside of scrolling container, just use scrollPaddingStart prop that the library provides like I asked you yesterday" - it only believed me more than its own hallucinated "evidence" was when I asked it to send me the screenshots about stuff being "fixed", "finished", and "done" and then I asked from my phone while on train something like "are you serious, did you even look at those screenshots?".
And it (re-)introduced ~4 different race conditions between scroll detection and/or programmatic scroll and/or rendering of auto-expanding elements and filters for deep links in various requestAnimationFrames, useEffects, event handlers, duplicated state management - not sure how many more it caught itself with its own sloppy unit and e2e playwright tests, but I when I was playing with Fable's capabilities instead of fixing the code manually, it was perfectly capable fixing each issue for which I gave it repro steps about stuff happening "sometimes" on a concrete screen/data combo (but finding those repro steps for non-deterministic race conditions is ~90% of the mental work anyway, fixing code is usually "just" mechanical at that point).
...tbh I used "a bit" more swearing in my actual prompts since it didn't take me seriously enough when I tried to tune down my language - turns out swearing at Fable was very useful when I later asked Grok 4.5 to analyze my prompts (with review by Fable and Sol) and the only common factor about times when I was not swearing turned out that I was not running the app at those times (== not manually testing it) - and the agentic advice about how to improve my CLAUDE.md/prompting I got in the report https://peter.hozak.info/claude/pr354-timeline.html#inflections turned out "meh" at best, none of the problems got much better when I tried another feature this month.
These days, I am trying to treat the coding agents swarm with Fable as orchestrator as if they had zero awareness that the state of the world and that it changes over time from their and other actions (or any applied-understanding of the concept of time at all), as if Church-Turing thesis was false and the LLMs operated purely on static functional input-output abstractions with huge gaps in their imperative intuitions (both their actions a la git feel like cargo culting, and any "reasoning" about state management code feels like they use words like "runtime" without having any good representation of the correct concept of "running", as if it was about the output of bash or github actions, as if there was some kind of metaphysical equivalence between "user clicking" and "playwright script in a file on disk as input and green/red as output").
- ^
the top being "Read a CSV with pandas, clean it up, group by something, and make a plot with matplotlib."
About a year ago, David and I put up two bounty problems involving natural latents. I am now about 80% confident that both have been resolved, both within the past couple months. Both cases made heavy use of LLMs and Lean.
The first to land was Grisha Pochuev's counterexample to the "Existence of a Deterministic Maximal Redund" conjecture. It's pretty readable, and I'm mostly convinced that it works. The original bounty post offered $500 for a proof or partial payout for a counterexample, with partial payout depending on how thoroughly the counterexample killed hope of any nearby variant of the conjecture. I think this counterexample is worth $300. Good job Grisha, and hopefully I can figure out a not-too-painful way to send you money.
Meanwhile, for a couple months David has been cranking away on "secret project X", with the promise that he'd tell me what the project was if and when it bore fruit. Well, apparently it bore fruit; he now has a proof that existence of a stochastic natural latent implies existence of a deterministic natural latent, which was our other bounty problem. The proof is apparently "pretty gnarly", lots of cases, all LLM-coded in Lean. I have not looked at the proof at all, but I'm operating on the assumption that it works and I'm hoping it will be simplified a lot in the coming weeks.
... and while all that was going on, I've spent the last few months mostly doing interp experiments. Some time early this year, Claude Code reached the point where it can handle my day-to-day interp coding needs well enough that I never need to write the code myself, which has been a qualitative jump in usefulness. Claude's interpretations of results and suggestions for next steps are still mostly useless, but it can at least write the code, and (I think) I can usually tell by looking at graphs/tables of outputs if the code is wrong.
So across the board, LLMs have started to meaningfully accelerate our work within the past ~4 months.
This is all in stark contrast to two years ago, when I reported that:
and:
At the time, multiple people hypothesized that we were just bad at using LLMs. Ray was one of those people; one day when we were coding something and the LLM was failing to help much, we invited Ray to take a look and hopefully tell us how to better use the LLMs. Ray concluded that our coding problems really were quite a bit more complicated than his day-to-day, and LLMs probably were not as good at them.
But that's in the past now. LLMs still do not look close to being able to do all the core pieces of my work, but they are at least accelerating meaningful parts in a big way, enough to qualitatively shift what we do and how we do it.