Mainly commenting on your footnote, I generally agree that it's fine to put low amounts of effort into one-off simple events. The caveat here is that this is an event that is 1) treated pretty seriously in past years and 2) is a symbol of a certain mindset that I think typically includes double-checking things and avoiding careless mistakes.
I don't know all the details of what testing was done, but I would not describe code review and then deploying as state-of-the-art as this ignores things like staged deploys, end-to-end testing, monitoring, etc. Again, I'm not familiar with the LW codebase and deploy process so it's possible all these things are in place, in which case I'd be happy to retract my comment!
I know this is going to come off as overly critical no matter how I frame it but I genuinely don't mean it to be.
Another takeaway from this would seem to be an update towards recognizing the difference between knowing something and enacting it or, analogously, being able to identify inadequacy vs. avoid it. People on LW often discuss, criticize, and sometimes dismiss folks who work at companies that fail to implement all security best practices or do things like push to production without going through proper checklists. Yet, this is a case where exactly that happened, even though there was not strong financial or (presumably) top down pressure to act quickly.
EDIT: I now see you research these questions and so want to add a disclaimer that I have not thought about these things nearly as deeply as you probably have...
Epistemic status: very speculative.
Cool post, I've long been fond of the, likely less difficult, thought experiment of whether we can grow a house using synthetic biology.
At first, I was thinking growing vs. building was just about the amount of labor involved to go from raw materials to final product. Then I realized this doesn't work because under this definition a fully automated robot factory would qualify as growing a car.
My next best guess is that "growing" is related to:
In terms of KPIs, the first things that come to mind are metrics like:
This and the linked post have been really helpful for my attempts to better internalize the Kelly Criterion. Thanks!
EDIT: I see you've corrected the mistake with the 12% and 2.7x return that I originally discussed below in a subsequent post so the details below aren't necessary. Maybe consider linking that post in the Addendum?
Mostly unrelated to the above, this is sort of a nitpick but between the body and the addendum, you (implicitly) switch from the odds-as-ratio-of-probabilities representation of to one in which is net fractional odds and is assumed to be . I know this makes sense because you're explicitly talking about bets in the final section but I'm bringing it up because it might throw off someone who hasn't read as many discussions of Kelly.
Another great example of this is Striped Smith-Waterman, which takes advantage of SIMD instructions to achieve a 2-8 speed-up (potentially much more on modern CPUs though) for constructing sequence local alignments.
As someone who has also struggled with similar issues, although in a different context than writing papers, I found some of the answers here helpful and could imagine some of them as good "tactical advice" to go along with cultural norms. I also ended up looking through Google's SRE book as recommended in Gwern's answer and benefited from it even though it's focused on software infrastructure. In particular, the idea of treating knowledge production as a complex system helped knock me out of my "just be careful" mindset, which I think is often one of the harder things to scale. Of course, YMMV.
What do you think about synthetic biology as a manufacturing technology?