This is a special post for quick takes by dr_s. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I've been thinking about paperwork. I actually work in a heavily regulated industry so I'm familiar with it first hand - as a civilisation, our agreed upon way of dealing with safety and regulated situations is "write it down". Obviously there is a usefulness of having a record of things after the fact if e.g. something goes wrong and you need to trace back what and when and because of who. But also, the documenting can go above and beyond that - we rarely just store a barebones bullet point list of things or a graph of relationships, which is the relevant information, we write full documents, sometimes in a special "document-y" language register that tends to be more redundant than commonly spoken English just to sound more official. And the result is often an aggregate amount of documents that can't ever really be read or fully taken in by anyone.
Now I'm worried LLMs might make this even worse by reducing the cost of producing text. This means if it's perceived that "more text = more safety/accountability" then we're more likely to produce even more documents as the LLMs can do it for us. This essentially means we give them the key information and then they uselessly expand it in a much bigger amount of data, which then odds are will be taken in by a second LLM and compressed back into the key information, hopefully faithfully, if anyone ever needs to go through it (there's already lots of AI startups doing things of this sort). This seems a lot more inefficient and noisy than the other way around, which would be I guess finding a way to store as much information as possible in a barebones format. Something like automated logging in software, but for an organisation/process.
Dunno, just throwing thoughts around, but what would a possible movement in the direction of distilling rather than inflating the amount of data we use for traceability and accountability look like?
There's a phenomenon that I think is quite common (and pernicious) but I don't know if it already has a name. I call it "suffocating lightning in a bottle". It's when a group of people does something cool and we as a civilization think "oh, that was amazing! Now if only we could make it systematic and scale it up, then it would be even better" and instead all we end up with is a bunch of dumb rules and bad incentives that suffocate rather than promote creativity, because creativity fundamentally resists that kind of systematization. It's a Molochian trap in which after you throw all the other values under the bus you don't even get the thing you wanted, just some useless Goodhart-ed metric. Examples of this: the Agile paradigm in software development and the modern system of peer-reviewed science.
In my opinion, the problem with Agile is that most companies adopt the keywords, but do not change their actual practices. Instead, they insist that they are giving new, more pragmatic meaning to the keywords. They would admit that they are not following the rules as they are defined in theory, because they think the theory is not realistic.
This sounds to me different from the situation in science, in the sense that there is no authoritative "Science Book" that the research companies would be clearly violating.
The problem with science seems like "suffocation by bureaucracy", the problem with agile seems like "using the keywords, without even trying to do the substance".
The problems can manifest a bit differently but I think the essence is the same. With Agile, it's "we want to emulate this success" without actually understanding any of the reasons of why it was a success. As you say, the focus is just on the buzzwords, but that's the point of the problem, that people focus on superficial details because the real essence of what made the original good thing tick was probably just "it was a lot of smart people working well together", and everything else was just superfluous or emergent detail. It's a cargo cult.
The situation with science I'm pointing at is the whole reproducibility crisis, publish-or-perish culture etc. So more like, academia/research really. In both cases the idea is trying to squeeze more productivity and accountability from the creative workers (developers/scientists) via systems. But in the end the systems become just stifling and pointless, or produce terrible incentives. The essence of what made the good stuff be good was that some people simply were able to do good work by freely exploring things they enjoyed to do and being left able to self-organize, and that isn't reproducible via rigid sets of rules. In fact very often the people who are good at doing the thing aren't necessarily very good at explaining what made them good at doing the thing (see: the authors of the Agile Manifesto who now mostly seem to have recused it).
Yeah, having the group of smart people who can work together is a crucial ingredient, and the attempts to replicate the success with different teams will fails miserably. And the next step is giving those people autonomy. What is the point of hiring people who are smart, good at their work, often obsessed with their work... and then having them micromanaged by someone who probably couldn't write a short shell script if their life depended on it?
The Scrum Guide is basically about how to get rid of managers, without everything falling apart. (All the bureaucracy introduced in Scrum was meant as a replacement for the usual company bureaucracy, not as an extra layer on top of it.) But companies somehow introduce "Scrum" while also keeping the managers and all the micromanagement, only now the developers have to do the extra daily stand-ups (where they do not literally stand up) and the ritual of estimating story difficulty (where the management has already decided that "this has to be done by the end of the month, because we have already promised that to the customer", and you are just choosing which tasks get done during the first two-week sprint, and which during the second one).
(This is a sensitive topic for me, because I actually have a Scrum Master training, and then I see how all companies talk about doing "Scrum" while often doing the exact opposite of it. Whenever a manager tells me that we need to do some pointless thing "because of Scrum", I suppress the urge to scream. But then, if you have a manager, you have already failed at Scrum. A good topic for the retrospective, except you probably don't have a retrospective, because the one thing the management hates is feedback provided in ways that are not under their control.)
And I guess it is similar in science. When you have smart people collaborating, great things happen, you just need to avoid some possible failure modes... such as people doing some esoteric thing for years with zero ability to communicate the meaning of it to the public (which pays the bills) and sometimes even to other scientists. But instead of keeping it at a reasonable level (such as: once in a year you have to publish an article describing your current progress, and I want other scientists' feedback on it) it becomes a game of maximizing the articles.
(Science is also broken on many levels. I know people who publish a scientific journal that contains some good articles and some bad ones. The good articles are worth it, but I asked why they also publish the bad ones, if they know they are bad? Their journal is not famous, so they do not get enough good articles. But why can't they simply publish less? Turns out, journals themselves are also somehow graded by how many articles they publish, so a journal that published only 10 good articles a year would get a worse rating than a journal that publishes 10 good articles and 20 bad articles a year! The numbers are not exact here, but this is the idea.)