In general the article was interesting. However the failures described seem to be related to lack of perspicacity more than to irrationality. If upper management are fooled by the middle managers’ signalling, they are not perspicacious enough and probably aren’t fit to be doing their jobs – training in rationality is unlikely to change that.
In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering. They haven’t taken one class, read one article, or been to one workshop. At best, they’ve managed other failing software projects.
This is not irrationality per se, but stupidity and incompetence.
Yes, businesses are under pressure to gravitate toward bad engineering practices, but shouldn’t they be under equal market pressure to compete against companies that are using actually good software engineering practices? Why sure, in the long run. But as Keynes succinctly put it, “In the long run, we’ll all be dead.” Eventually is a long time.
An alternative perspective is that extensive government intervention substantially reduces the market pressure that large companies experience. Here is Moldbug on the subject of Dilbertization (the government intervention he implies is its propping up of the maturity transforming banking system):
The late Communist world was a world in which it was often your job to do strange, useless things badly, all day, for no good reason at all.
There is another world in which it is often your job to do strange, useless things badly, all day, for no good reason at all. This is the world of Dilbert. Many of us have experienced it.
My theory is that Dilbert and Brezhnev are the same thing. I call it Dilbert-Brezhnev syndrome, or DBS. While we are certainly not the Soviet Union, my theory is that America has contracted a rather serious case of DBS.
The Soviet Union was a world in which business bore no relation to profit. People did strange, useless things badly because, lacking the discipline of profit that enforces efficiency, they succumbed to ulterior motives. Their unprofitable enterprises, purportedly businesses, were in fact patronage structures.
America is a much more interesting case, because (aside from its endlessly burgeoning political system, including its grant-funded "nongovernmental" periphery), the industries in which we see Dilbert syndrome are private, profitable. Nothing in America today is Brezhnev bad, but it is getting there. [...]
The answer, I think, is our friend zombie finance. We do not have Gosplan, but we have Wall Street. America is Dilbertized to the extent that Wall Street is zombified.
Let's take the loans that created the housing bubble. These were zombie loans to a T. So, for example, Steve Sailer asks: where was capitalism? Why wasn't anyone betting against these loans, and driving them down to their true value?
The answer is that the free market bears very little relationship to the process that distributed these loans. [...]
My feeling is that American zombie finance is largely responsible for the appearance of DBS in the New World. Why is the country covered with hideous developments, strip-malls and chain stores? Because it has, or had, a financial system designed to finance these things. Many of us would prefer a Ritual to a Starbucks and a Joe's Diner to a Burger King, but the chains have an unbeatable advantage: it is much easier for them to get a loan. [...]
Starbucks is subject to the discipline of profit - but it is not as subject as it should be, because its sheer size gives it access to zombie money. Thus the generic can defeat the specific, blandness outcompetes character, and we drink charred cat hair rather than coffee.
That's true... for the banking sector. However, the author was talking about the software projects in general. In my experience (and the author's experience appears to agree with mine) the sort of organizational irrationality peculiar to software isn't especially overrepresented in any particular sector. It's present in all sectors, from banking to video games. There's a deep intuition suggesting that adding more workers to the project will make progress occur more quickly. (Bad) middle managers play to that intuition and add workers even when the addition of more workers actually slows down the progress of the project.
I haven't seen any evidence of extensive governmental intervention in, say, XBox Games, but management practices at EA appear to fit this stereotype to a tee.
And Microsoft's behavior on device drivers, as explained in the second 'Halloween document': close the process on driver development so you can assert more control over them so you can improve their quality... but instead it gets to the point where major electronics corporations have to wing it on making device drivers because the process is so closed. And then everything breaks harder than before.
That's true... for the banking sector. However, the author was talking about the software projects in general.
Missed the point completely I'm afraid.
ETA: because clearly Moldbug is talking about government interference in the banking system disrupting the free market in general, i.e. including the software industry.
He could be talking about that, but he attributes all lasting business irrationality to government interference. There are plenty of other sources.
Yes, there are many other ways in which (according to Moldbug et al) government interference causes business irrationality. After all, Moldbug's favourite thinker Thomas Carlyle popularised the term "red tape" as a metaphor for bureaucratic procedure.
However, the excerpt that I quoted deals exclusively with the idea that government intervention (usually disguised through the concept of loan guarantees, but overt in times of recession) in the banking system causes business irrationality, because if the banking system isn't subject to the discipline of profit then this lack of profit-discipline contaminates the private sector via the loans that the banking system provides.
I was disappointed that user:quanticle was upvoted for responding, "That's true...for the banking sector" - this suggests that he didn't make a reasonable attempt to understand the excerpt, because although readers are welcome to disagree with it in an intelligent and thoughtful way, clearly his interpretation is simply an error of comprehension.
That's true... for the banking sector. However, the author was talking about the software projects in general.
The banking sector interacts with all of private industry, therefore government interference in the banking sector (arguably) causes Dilbertization in software engineering and elsewhere. The articles that I linked to explain this idea in more detail.
I prefer this explanation to the idea that “eventually is a long time”, because it seems to me that a free market should punish wanton incompetence efficiently enough such that Dilbertization doesn’t become a commonplace state of affairs. There is ample opportunity for profit to be made in eliminating incompetent management before the point at which a company implodes; widespread Dilbertization (in which, as the writer of your article described, patently inept managers preside over multiple failed projects) implies to me that what is really happening is that the profit motive in eliminating this incompetence just isn’t that strong – in which case Moldbug’s explanation fits the facts better.
Other forms of government intervention (e.g. “red tape”) also cause Dilbertization, but influence through the maturity mismatched banking system is particularly insidious. Government loans are disguised as “deposit insurance”, propping up an unstable fractional reserve system that wouldn’t exist in a free market context, and when the system inevitably crashes private industry can be openly Sovietised on the basis that the (allegedly) free market has failed – a “piss on my boots and tell me its raining” scenario.
In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering. They haven’t taken one class, read one article, or been to one workshop. At best, they’ve managed other failing software projects.
One might be the most dangerous number of books for a manager to have read.
Good point...
I found many of the books I read quite bad, full of hot air etc.; there's a few lessons to be learned from the classics (like Brooks etc.), but I'm not really convinced about many of the new stuff (agile, tdd etc), after seeing it in practice. It seems a combination of good existing ideas + ideas which are unproven at best.
My experience is that the majority of such signalling is around the time that will be needed. You look at a new project and think to yourself it's probably about 6 months work. Your manager tells you that it "has to" go live in 2 months. And then somehow you end up saying "Okay, we'll try" instead of "That seems very unlikely". Even if the previous overdue project was quite similar.
Technical people are not comfortable asserting a realistic schedule to management and management is not comfortable asserting it to shareholders.
My managers, therefore quite rationally did things that maximized their apparent value in the eyes of their bosses, even if it meant that the project (and, as a result) the entire organization was hurt.
The problems described are applicable to corporations and collective action in general.
The book Moral Mazes chronicles a sociological study in the 1980s, where a sociologist spent years living with the savages of corporate bureaucracy, learning their customs, behaviors, and morals. The irrelevancy of truth, the shifting of blame, the feudal relationship to those higher up, the obligation to keep knowledge from your bosses so that they could avoid responsibility, the situational ethics totally contradicting everything they otherwise profess in life - all of it there, and fully documented in the study.
Orwell and Hayek had it right. Organizations will get predictable results depending on how knowledge, power, and responsibility are aligned despite the clear global suboptimality (read abomination) of the solution to the individuals involved. Align those things wrong, and you get Big Brother and a boot stomping a human face, forever. Align them differently, and you get the modern corporation, or government bureaucracy, or or open source and maker movements.
Why Software Projects are terrible and how not to fix them (by Drew Crawford):
Unless you are having a meeting with the one person who is going to use the software that you’re writing, you’re not meeting with the real customer. You’re meeting with a person who has to explain to someone who can explain to someone who can explain what you’re saying to the real customer. It’s not enough to convince the person you’re sitting in the room with that Agile is a good idea. He has to convince his boss. That person has to convince his boss. That person has to convince the sales team. The sales team has to convince the customer. If the customer is b2b, your contact at the customer organization has to convince his boss. Who convinces his boss. Who convinces the real customer. Maybe. Unless that sale is also b2b. This is a very long game of telephone. If the guy you’re talking too is thinking “This sounds like a really good idea but I’m concerned I can’t sell this upstairs,” you are dead in the water. At any point in the chain, if somebody thinks that, you are dead in the water. You can’t just say “It’s objectively better,” you have to show how he can turn around and sell the idea to someone else.
Put yourself in the middle manager’s shoes. If the project goes bad, he has to “look busy”. He has to put more developers on the project, call a meeting and yell at people, and other arbitrary bad ideas. Not because he thinks those will solve the problem. In fact, managers often do this in spite of the fact that they know it’s bad. Because that’s what will convince upper management that they’re doing their best.
In other words, it's all about signaling, isn't it? Managers will take actions that actively harm the continued progress of the project if that action makes them look "decisive" and "in charge". I've seen this on many projects I've been on, and it took me a while to realize that my managers weren't stupid or ignorant. It's just that the organization I was working in put a higher priority on process than on results. My managers, therefore quite rationally did things that maximized their apparent value in the eyes of their bosses, even if it meant that the project (and, as a result) the entire organization was hurt.
Crawford then goes on to detail why organizations with such maladaptive practices survive:
Yes, businesses are under pressure to gravitate toward bad engineering practices, but shouldn’t they be under equal market pressure to compete against companies that are using actually good software engineering practices? Shouldn’t, at some point, bad companies simply implode under their own weight? Why sure, in the long run. But as Keynes succinctly put it, “In the long run, we’ll all be dead.” Eventually is a long time. It’s months, years, or decades. A project can be failing a long time before management is clued in. And even longer before management’s management is clued in. And it can be ages before it hits the user.
I think this is something that we as rationalists sometimes forget about. Irrationality has momentum. Humans have been thinking intuitively for thousands (hundreds of thousands, even) of years before we figured out how to think with rigorous rationality. Even if rationality had a massive advantage of intuitive thinking in everyday situations (it doesn't, as far as I can tell) it would take a very long time for rational thought to propagate through society.
So the next time you get frustrated at some bit of wanton irrationality, remind yourself, "Momentum," before you get frustrated.
EDIT: Fixed spelling as per RolfAndreassen's post.