Huh. Interesting. Eliezer seems to imply, in that post, that individuals aren't as susceptible to this as organizations. Maybe it's just harder to notice?
Individuals are indeed just as susceptible. And it is indeed harder to notice, just like an organization as a whole rarely notices its own lost purpose. Taking instrumental goals for terminal values is a common pattern.
"The Solution" depends on "The Problem", which depends on "Whose Problem?"
I'll talk corporate bureaucracy.
Organizations are made of people who are acting to solve their own problems, which are primarily their personal power, security, and status. Even if there were precise agreement on what the "organization's goals" are, aligning individual incentives with the organization's goals is extremely hard, even in the broadest sense.
"Our people are our greatest asset." Probably. But they're also parasites feeding as well as they can on the organizational host.
That's just the incentive problem. There's also the knowledge problem.
Individuals can lose sight of their purpose as well, but a person has more coherent interests and knowledge.
Individuals can lose sight of their purpose as well, but a person has more coherent interests and knowledge.
This. It is easier to lose the sight of a goal, if you were never told about it, or if it is not actually your goal.
Sometimes people forget their own goals, just like described in this article, but forgetting someone else's goal is much easier.
My now emperor was asked by his people to improve the condition of things (as things were apparently terrible), and his response was that he needed to conquer the rest of Asia first, to ensure their security.
This reminds me of:
I have no way to defend my borders but to extend them.
Empress Catherine the Great of Russia
Summary: It's easy to get caught up in solving the wrong problems, solving the problems with a particular solution instead of solving the actual problem. You should pay very careful attention to what you are doing and why.
I'll relate a seemingly purposeless story about a video game to illustrate:
I was playing Romance of the Three Kingdoms some years ago, and was trying to build the perfect city. (The one city I ruled, actually.) Enemies kept attacking, and the need to recruit troops was slowing my population growth (not to mention deliberate sabotage by my enemies), so eventually I came to the conclusion that I would have to conquer the map in order to finish the job. So I conquered the map. And then the game ending was shown, after which, finally, I could return to improving cities.
The game ending, however, startled me out of continuing to play: My now emperor was asked by his people to improve the condition of things (as things were apparently terrible), and his response was that he needed to conquer the rest of Asia first, to ensure their security.
My initial response was outrage at how the game portrayed events, but I couldn't find a fault in "his" response; it was exactly what I had been doing. Given the rest of Asia, indeed the rest of the world, that would be exactly what I would have done had the game continued past that point, given that threats to the peace I had established still existed. I had already conquered enemies who had never offered me direct threat, on the supposition that they would, and the fact that they held tactically advantageous positions.
It was an excellent game which managed to point out that I have failed in my original purpose in playing the game. My purpose was subsumed by itself, or more particularly, a subgoal. I didn't set out to conquer the map. I lost the game. I achieved the game's victory conditions, yes, but failed my own. The ending, the exact description of exactly how I had failed and how my reasoning led to a conclusion I would have dismissed as absurd when I began, was so memorable it still sticks in my mind, years later.
My original purpose was subsumed. By what, exactly, however?
By the realities of the game I was playing, I could say, if I were to rationalize my behavior; I wanted to improve all the cities I owned, but at no point until I had conquered the entire map could I afford to. At each point in the game, there was always one city that couldn't be reliably improved. The AI didn't share my goals; responding to force with force, to sabotage with sabotage, offered no penalties to the AI or its purposes, only to mine. But nevertheless, I had still abandoned my original goals. The realities of the game didn't subsume my purpose, which was still achievable within its constraints.
The specific reasons my means subsumed my ends may be illustrative: I inappropriately generalized. I reasoned as if my territory were an atomic unit. The risks incurred at my borders were treated as being incurred across the whole of my territory. I devoted my resources - in particular my time - into solving a problem which afflicted an ever-decreasing percentage of that territory. But even realizing that I was incorrectly generalizing wouldn't have stopped me; I'd have reasoned that the edge cities would still be under the same threat, and I couldn't actually finish my task until I finished my current task first.
Maybe, once my imaginary video game emperor had finally finished conquering the world, he'd have finally turned to the task of improving things. Personally, I imagine he tripped and died falling down a flight of stairs shortly after conquering imaginary-China, and all of his work was undone in the chaos that ensued, because it seems the more poetic end to me.
A game taught me a major flaw in my goal-oriented reasoning.
I don't know the name for this error, if it has a name; internally, I call it incidental problem fixation, getting caught up in solving the sub-problems that arise in trying to solve the original problem. Since playing, I've been very careful, each time a new challenge comes up in the course of solving an overall issue, to re-evaluate my priorities, and to consider alternatives to my chosen strategy. I still have something of an issue with this; I can't count the number of times I've spent a full workday on a "correct" solution to a technical issue (say, a misbehaving security library) that should have taken an hour. But when I notice that I'm doing this, I'll step away, and stop working on the "correct" solution, and return to solving the problem I'm actually trying to solve, instead of getting caught up in all the incidental problems that arose in the attempt to implement the original solution.
ETA: Link to part 1: http://lesswrong.com/lw/e12/subsuming_purpose_part_1/