One thought is that, when the evaluator of a team is also a member of the team (e.g. the leader), everyone on the team is incentivized to appease the evaluator to get a good evaluation. This makes the leader/evaluator very biased in favor of individuals who befriend him/her. It also makes everyone biased in not just cooperating with the leader/evaluator (which is good in a team), but always agreeing with the leader/evaluator (which is bad in any team responsible for decision making).
Having an external system to evaluate members of a team can reduce how much evaluation the leader does, and reduce these perverse incentives.
It might be especially helpful for decision making teams. Unfortunately it's also especially difficult to evaluate members of a decision making team using AI, because AI sucks at decision making, and decision making is a skill where "only people with the skill can judge others' skills."
This is a real dynamic that Conway will have to carefully design incentives around — one intermediate solution is to attribute reward and blame to actions instead of individuals. There can still be internal debate over whose responsibility the actions themselves were, which will likely continue to be political and outside of Conway's scope. I think it's reasonable to believe that companies will be penalized for unruly internal politics more now than ever before, but also to concede that some internal politics will probably always persist.
Oh, I think I read your post without understanding what "credit attribution" meant.
There are indeed two completely distinct problems of "evaluating decisions by attributing credit to them," and "evaluating individuals by attributing credit to them." I somehow assumed the latter interpretation when you were talking about the former the whole time. The words "credit" and "attribution" usually refer to individuals, and I also wasn't reading closely.
Never mind.
Just read When Money is Abundant, Knowledge is the Real Wealth. Excellent argument for companies as learning machines, not money making machines.
The purpose of a company is to learn about its environment.[1] [2] Companies are learning machines, and the best ones focus on extracting more information with less noise as fast as possible. This dynamic has always been true, and promises to become even more true because smaller teams increasingly generate outsized returns. This rising competitive pressure means organizations have an ever-shrinking margin for error in their information uptake rates. If you’re planning on participating in the new world, the feedback loop is your primary product.[3]
Companies waste their data, and it’s not for lack of trying. Companies spend billions of dollars extracting and storing information about user interactions. But at the end of this effort they are left with a few colorful graphs designed to persuade, not to inform. And it’s not their fault. Companies are bottlenecked by intelligence. Learning from data has been a hugely expensive effort until recently because of labor costs. “Until recently” because frontier models solve the cost of labor problem.
In other words, companies are learning machines, and now it’s possible for the first time to learn at scale.
Imagine you’re playing chess, you don’t know the rules, and you only get one bit of feedback: whether you won or lost at the end of the game. Over 1,000 iterations of the game, you may improve by classifying every sequence of actions that led to a winning strategy as “good” and every sequence of actions that led to a losing strategy as “bad.” This is a black box RL approach, and the simplest version of credit assignment. However, if you have the ability to figure out why you won or lost, you will become a better chess player in a smaller number of iterations.[4]
Now imagine you’re running a company. You’re operating in an exponentially more complex environment. It’s not a viable strategy to call the sequence of actions that led to the company going bankrupt “bad”. Instead, we can model the environment itself as a white box that can be cracked open and analyzed. In other words, we’re trading off computational requirements for state of the art sample efficiency. But that’s fine, because the cost of intelligence is dropping by an order of magnitude every year. If indeed a company’s purpose is to learn about its environment, then a company’s purpose is to do credit attribution.[5]
Companies are learning machines, it's possible for the first time to learn at scale, and the way to learn at scale is to build credit attribution systems.
This new world will move at a fast clip. Every learning will be extracted and operationalized in real time. Company operations will start to resemble something more akin to high frequency trading or military operations rather than the paper-pushing politics of companies today.
Data-rich regimes with fast and complex decision-making requirements have a common interface pattern: a centralized operating picture that combines observability, reasoning, and action in a single place. Traders have Bloomberg terminals, and military operations have command and control systems (see Anduril's Lattice OS). Conway (my company) is building a command and control system for companies, powered by a credit attribution engine.
Conway’s law suggests that the outputs of a company inevitably mirror how the company is internally structured. Companies who refuse to learn have always suffered, and now they’ll be saved from their suffering by the hands of sweet death. The converse is also true. Companies that choose to embrace their roles as learning machines may join the first rarified cohort of companies that last.
If you want to talk more about this, I'd love to chat. You can learn more about me here, email me at anne@conway.ai, or message me directly.
See Patrick McKenzie's discussion here.
Isn't a company's purpose to make money? Of course. But if you needed to pinpoint the guiding principle of a company, it's to build a machine that makes money, not to opportunistically make money. This is the difference between winning the battle of a favorable quarter and winning the war of conquering a market. How do you build a machine that makes money? You learn about your environment, and you learn about your environment better than your competitors. Companies are learning machines.
Inspiration for this language comes from Raemon's post a while back Feedback-Loop First Rationality.
Great grounding from Andrei Karpathy's work Deep Reinforcement Learning: Pong from Pixels.
If you want to read more about credit attribution, especially in a company context, read Gwern's Evolution as a Backstop for Reinforcement Learning.