I really enjoyed this study. I wish it weren't so darn expensive, because I would love to see a dozen variations of this.

I still think I'm more productive with LLMs since Claude Code + Opus 4.0 (and have reasonably strong data points), but this does push me further in the direction of using LLMs only surgically rather than for everything, and towards recommending relatively restricted LLM use at my company.

It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?"

At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.

There was a serious bug in this post that invalidated the results, so I took it down for a while. The bug has now been fixed and the posted results should be correct.

One sort-of counterexample would be The Unreasonable Effectiveness of Mathematics in the Natural Sciences, where a lot of Math has been surprisingly accurate even when the assumptions where violated.

Unfortunately, I think it isn't very compatible with the way management works at most companies. Normally there's pressure to get your tickets done quickly, which leaves less time for "refactor as you go".

I've heard this a lot, but I've worked at 8 companies so far, and none of them have had this kind of time pressure. Is there a specific industry or location where this is more common?

Cooking: 

• Smelling ingredients & food is a good way to develop intuition about how things will taste when combined
• Salt early is generally much better than salt late

Data Science:

• Interactive environments like Jupyter notebooks are a huge productivity win, even with their disadvantages
• Automatic code reloading makes Jupyter much more productive (e.g. autoreload for Python, or Revise for Julia)
• Bootstrapping gives you fast, accurate statistics in a lot of areas without needing to be too precise about theory

Programming:

• Do everything in a virtual environment or the equivalent for your language. Even if you use literally one environment on your machine, the tooling around these tends to be much better
• Have some form of reasonably accurate, reasonably fast feedback loop(s). Types, tests, whatever – the best choice depends a lot on the problem domain. But the worst default is no feedback loop

Ping-pong:

• People adapt to your style very rapidly, even within a single game. Learn 2-3 complementary styles and switch them up when somebody gets used to one

Friendship:

• Set up easy, default ways to interact with your friends, such as getting weekly coffees, making it easy for them to visit, hosting board game nights etc.
• Take notes on what your friends like
• When your friends have persistent problems, take notes on what they've tried. When you hear something they haven't tried, recommend it. This is both practical and the fact that you've customized it is generally appreciated

Conversations:

• Realize that small amounts of awkwardness, silence etc. are generally not a problem. I was implicitly following a strategy that tried to absolutely minimize awkwardness for a long time, which was a bad idea

• using vector syntax is much faster than loops in Python

To generalize this slightly, using Python to call C/C++ is generally much faster than pure Python. For example, built-in operations in Pandas tend to be pretty fast, while using .apply() is usually pretty slow.