From my humble experience, I've come to believe that Deming's view was the wisest:
The idea of a merit rating is alluring. The sound of the words captivates the imagination: pay for what you get; get what you pay for; motivate people to do their best, for their own good. The effect is exactly the opposite of what the words promise. Everyone propels himself forward, or tries to, for his own good, on his own life preserver. The organization is the loser.
Thanks for taking the time to transparantly writing down your approach. I'm spending more and more time optimizing developer effectiveness at work, so posts like this may help me in my own behavior.
He argues that "interdependencies and nuances of individual work are too complex to be measured by an outside observer," concluding that the only good way to do it is to measure it at the team level -- "does this team consistently produce useful software on a timescale of weeks to months?"
I agree; what other people do can have a strong impact on your productivity, and what you do can have a strong impact on their productivity. Probably the worst case is a policy where you fire the least performing member of each team, because that directly incentivizes doing things that slow down others -- like, forget that anyone working in such environment would ever document anything in an actually useful way.
Makes me wonder... did any company try to internally implement "group selection"? I mean some policy like: keep the best teams intact (even if some of their members are low-performing), and fire the worst teams en masse (including their high-performing members)? Assuming that the teams work on independent products, without the opportunity to sabotage each other. I wonder what teams would survive such selection...
Or perhaps a more flexible version of the above, where the high-performing teams would be allowed to fire some of their members, if the majority of team members vote (anonymously?) for that. If the team is aware that next year they will be judged by the group performance again, they have an incentive to fire an obvious slacker, but wouldn't fire a person who may seem replaceable to the management, but the team perceives them as important for the group dynamic. That would allow the team to have people specialized on doing things that seem useful to fellow developers but unimpressive to the management. Also, the team would have to approve a new hire. (New hires are necessary, because the existing team members sometimes leave the company on their own.)
On the other hand, when the low-performing teams are disbanded, their members could get a second chance, by receiving an invitation into a surviving team, or being randomly mixed into new teams. Only if your team is disbanded twice in a row, and no existing team wants you to join them, then you get fired.
(Perhaps in a sufficiently large company, the successful teams should be allowed to grow and eventually split into two teams, by their own internal decision. I say "sufficiently large", so that splitting the team will not feel like creating a competitor that may get my half of the team fired in turn. Like, if the bottom 10% teams are disbanded each year, and only the top 30% are allowed to split, that might make them feel sufficiently safe about the outcome. And maybe there could be some encouragement to splitting as opposed to playing it safe. For example, a certain fraction of company profits could be distributed as a bonus to members of the teams created by splitting during the following three years, assuming they don't fall to the lowest 10% and get disbanded. -- Would need to think more carefully about potential perverse incentives this might create.)
I feel like this might prevent some types of careers like people who make the rest of the team suffer, but the management likes them, attributes the successes to them and the failures to other team members; or people who join as experts, introduce some random changes that seem like huge improvements to the management, and then leave before the project falls apart, being introduced to the new company as the experts who saved the previous company.
Also, it seems to me that companies often do the oppposite of this strategy -- successful teams complete their projects, and get disbanded as a consequence. No matter how well the employees cooperated with each other, they get randomly assigned to either the remaining or new projects, and the successful team dynamic is just thrown away. (Sometimes this makes the members of the formerly successful team quit, because they just lost something that kept them working there. Hey, if you are going to have a new project and new colleagues anyway, why not also try a new company that offers you a higher salary?) It is the teams failing to complete their projects that persist.
I'm going to call the type of management where codifying the steps to a workflow is a goal in and of itself the "procedure" orientation. There are managers at my job who are like this; they do not care in any meaningful way about outcomes or even if the procedure is actually achievable, considering the establishment and verification of procedure to be the desired end state. The charitable interpretation of this is that they are looking for legibility (they need to be able to see what is going on) and consistency (everyone needs to do the same thing).
My first impression is that when shifting from a results orientation or a procedure orientation to a process orientation, the overwhelming factor will be how well the translation from results to decisions goes.
As a note, in my experience people of the procedure orientation refer to themselves as process people. We're going to need a different name in order to keep this from being consistently misunderstood or co-opted by them.
I nominate: decision process orientation
Or simpler: decision orientation
There's a certain type of workplace where the only thing that matters is that you, as an individual, produce results that are valuable to the company. Netflix is famously like this (or claims to be); here's a quote from their culture page:
Succeeding on a dream team is about being effective, not about working hard. Sustained “B” performance, despite an “A” for effort, gets a respectful severance package. Sustained “A” performance, even with a modest level of effort, gets rewarded. Of course, to be great, most of us have to put in considerable effort, but hard work and long hours is not how we measure or talk about a person’s contribution.
There are a lot of ways this can be implemented. The "piece work" version (pay the worker a fixed wage per item produced) is traditional for manufacturing jobs. But for intellectual work, it is far harder to measure results on an individual basis. Let's take software engineering as an example. Maybe your boss wants you to build some features -- those features being shipped would be the result. But there are a lot of different ways you could build the features -- you could cut corners and introduce a lot of tech debt, or you could polish them for a long time. How is the boss going to evaluate your results? They would need to specify ahead of time the expected quality of the work, but specifying it precisely enough is about equivalent to doing the job -- so that won't work.
Isaac Lyman at the Stack Overflow blog has a piece about measuring developer productivity. He argues that "interdependencies and nuances of individual work are too complex to be measured by an outside observer," concluding that the only good way to do it is to measure it at the team level -- "does this team consistently produce useful software on a timescale of weeks to months?"
I suspect that of all their talk about measuring results, Netflix is probably (at the individual level) doing something more like what Lyman suggests -- using fuzzier methods to evaluate individual performance to decide whether to fire someone. (If anyone reading this works at Netflix, I would like to know whether I am right about this.)
Let's take a look at individual-evaluation methods. How should we decide if an individual is doing their part on a software engineering team? We can't measure something like hours worked, or lines of code committed; these measurements create bad incentives and fail in the obvious ways.
My ideal method, which I call "process orientation," would be to look at the work-related decisions the person made. As many decisions as possible, spanning all aspects of the work life -- should we refactor this component now or not? what should we name this method? is this well tested enough to ship? should I spend more time trying to figure out the best way to build this before diving in? how should I give feedback to my coworker about their bad code? should I quit the Slack app so I can focus, or leave it on in case someone needs me?
The next step is to ask, for each decision, if it was the right decision at the time they made it, given their knowledge and abilities. Hold people to the standard of (usually) making good decisions ex ante.
Note that this process does not take into account the outcome of the decisions! In some sense, this is the opposite of results orientation. In results orientation, you look at outcomes and ask "is this good or bad?" and then you reward or punish. In process orientation, you look only at their process of making decisions, and ask "does this process usually produce the best outcomes?"
My colleague Drew always says, "judge the shot while the ball is in the air." This basketball metaphor is a great one: most shots don't go in, so it is silly to spend much energy on whether or not a given shot made it. Basketball is all about whether the player positioned themselves appropriately to get open, used good form while shooting, and so on. Everything in human endeavor is like this -- you win some, you lose some; what matters is whether you made the best decisions at the time.
Now, process orientation doesn't mean we never look at the results -- it just constrains how you look at them. We look at results to figure out which decision-making processes are best, and over time, that feeds back into evaluating someone's performance. For example, perhaps we as a company realized that shipping bad code causes problems later on, so we decided code reviews need to happen on time, and communicated that. If the person continues deciding not to prioritize code reviews even after we've changed the process, it is likely bad decision making; but maybe they have an overriding reason.
Process orientation also does not mean that the company should bend over backwards to codify processes. Usually, we should limit ourselves to explaining best practices, then trust people's ability to make good decisions in the moment, because it's very hard to write down everything that matters in a decision. A prominent example would be COVID vaccine approvals, post trial. The AstraZeneca vaccine is still not approved in the US (as of this writing); but it is known to be safe and effective in the EU. It is not approved yet only because the written-down process doesn't have a good way to accelerate vaccine approvals in this sort of situation.
I bring up the vaccine example not to get in a dig at the FDA, but to point out that it is easy to take the idea of "process" in the wrong direction. Lots of companies have tons of internal bureaucracy about how to get things done. I think people should not be expected to violate bureaucratic rules in order to succeed at their job, and the best way to achieve that is to not create lots of red tape in the first place. Process orientation is about individual decision-making, not written-down process.
I think one of the games that best illustrates process orientation is high level poker. (Again, not really something I play myself, but I like watching it.) Great poker players all have a very disciplined process for which hands to bet preflop, calculating pot odds and deciding when to call vs. fold. You may go on a long winning or losing streak, but the virtue is sticking to your process -- if you get too outcome-oriented, that's called "going on tilt" and probably means you are about to lose a lot of money.
I'm a manager and I implement this system, and encourage others to do it too. Most feedback I give my reports is based on processes that they haven't perfected yet. Here are some ideas on ways to do this:
The book Principles by Ray Dalio is a wonderful resource on process orientation.
Last thing. I want to be really clear on this: the only purpose of good process is to produce good outcomes. A process is not good unless it produces good outcomes. I'm trying to walk a fine line in this post -- don't over-focus on the outcomes, but also don't attach virtue to process for its own sake.