Review
Review
Leading The Parade
57nim
1the gears to ascension
39aysja
8kave
4Alexander Gietelink Oldenziel
8ryan_greenblatt
10Alexander Gietelink Oldenziel
6kave
27Jozdien
13johnswentworth
8Alexander Gietelink Oldenziel
13NicholasKees
9Donald Hobson
4johnswentworth
8Paradiddle
4ryan_b
8abramdemski
4johnswentworth
5ryan_b
3N1X
8[anonymous]
6Thomas Kwa
6Elizabeth
5Morpheus
4Ben
4KanizsaBoundary
4tailcalled
3mattmacdermott
8Garrett Baker
7mattmacdermott
5johnswentworth
1Review Bot
New Comment
If you're talking about literal parades -- I lead them annually at a smallish renaissance fair. Turns out that people with the combination of willingness to run around in front of a group looking silly, and enough time anxiety to actually show up to the morning ones, are in short supply.
That parade goes where I put it. There are several possible paths through the faire and I choose which one the group takes, and make the appropriate exaggerated gestures to steer the front of the crowd in that direction, and then the rest follow.
I also play a conspicuous looking instrument in the parade at a small annual local event that we convene a "band" for, as well. Since the instrument is large and obvious, I'm typically shoved to the front of the group as we line up. I'm pretty sure that if I went off script and took the parade out of the gathering's area, they'd probably follow me, because nobody else is quite sure where we're supposed to be going. If I conspired with the other musicians to take the group out of the event, we could almost certainly make that happen. I'm curious how far down the road we could get the dancers following the parade before they realize something is amiss, but also really don't want to be the individual to instigate that sort of experiment.
Back in high school, I did marching band, I think if our leader had been misinformed about where we should go, we would have followed them anyway. That's mostly because marching band has an almost paramilitary obedience theme going on, and can get a bit culty about directors or leaders in my experience. Marching as a group also confers a certain immunity to individual responsibility as long as you're following your orders. There's this confidence that if the leader takes the group off course, that leader will be the only individual who's personally in trouble for the error. The group might get yelled at collectively for having followed, but no one person in the group is any more responsible for the error than any other, except for the leader.
From these experiences, I'd speculate that the reason we don't see literal parades being counterfactually led off course like that on a regular basis is because the dynamic of leading it disincentivizes abusing that power. Being chosen and trusted by a group to lead them in a public setting where any errors you make will be instantly obvious to all onlookers confers a powerful desire to not mess up.
8I do think that counterfactual impact is an important thing to track, although two people discovering something at the same time doesn’t seem like especially strong evidence that they were just "leading the parade." It matters how large the set is. I.e., I doubt there were more than ~5 people around Newton’s time who could have come up with calculus. Creating things is just really hard, and I think often a pretty conjunctive set of factors needs to come together to make it happen (some of those are dispositional (ambition, intelligence, etc.), others are more like “was the groundwater there,” and others are like “did they even notice there was something worth doing here in the first place” etc).
Another way to say it is that there’s a reason only two people discovered calculus at the same time, and not tens, or hundreds. Why just two? A similar thing happened with Darwin, where Wallace came up with natural selection around the same time (they actually initially published it together). But having read a bunch about Darwin and that time period I feel fairly confident that they were the only two people “on the scent,” so to speak. Malthus influenced them both, as did living in England when the industrial revolution really took off (capitalism has a “survival of the fittest” vibe), so there was some groundwater there. But it was only these two who took that groundwater and did something powerful with it, and I don’t think there were that many other people around who could have. (One small piece of evidence that that effect: Origin of Species was published a year and a half after their initial publication, and no one else published anything on natural selection within that timespan, even after the initial idea was out there).
Also, I mostly agree about Shannon being more independent, although I do think that Turing was “on the scent” of information theory as well. E.g., from The Information: “Turing cared about the data that changed the probability: a probability factor, something like the weight of the evidence. He invented a unit he named a ‘ban.’ He found it convenient to use a logarithmic scale, so that bans would be added rather than multiplied. With a base of ten, a ban was the weight of evidence needed to make a fact ten times as likely.” This seems, to me, to veer pretty close to information theory and I think this is fairly common: a few people “on the scent,” i.e., noticing that there’s something interesting to discover somewhere, having the right questions in the first place, etc.—but only one or two who actually put in the right kind of effort to complete the idea.
There’s also something important to me about the opposite problem, which is how to assign blame when “someone else would have done it anyway.” E.g., as far as I can tell, much of Anthropic’s reasoning for why they’re not directly responsible for AI risk is because scaling is inevitable, i.e., that other labs would do it anyway. I don’t agree with them on the object-level claim (i.e., it seems possible to cause regulation to institute a pause), but even if I did, I still want to assign them blame for in fact being the ones taking the risky actions. This feels more true for me the fewer actors there are, i.e., at the point when there are only three big labs I think each of them is significantly contributing to risk, whereas if there were hundreds of leading labs I’d be less upset by any individual one. But there’s still a part of me that feels deontological about it, too—a sense that you’re just really not supposed to take actions that risky, no matter how inculpable you are counterfactually speaking.
Likewise, I have similar feelings about scientific discoveries. The people who did them are in fact the ones who did the work, and that matters to me. It matters more the smaller the set of possible people is, of course, but there’s some level upon which I want to be like “look they did an awesome thing here; it in fact wasn’t other people, and I want to assign them credit for that.” It’s related to a sense I have that doing great work is just really hard and that people perpetually underestimate this difficulty. For instance, people sometimes write off any good Musk has done (e.g., the good for climate change by creating Tesla, etc.) by saying “someone else would have made Tesla anyway” and I have to wonder, “really?” I certainly don’t look at the world and expect to see Teslas popping up everywhere. Likewise, I don’t look at the world and expect to see tons of leading AI labs, nor do I expect to see hundreds of people pushing the envelope on understanding what minds are. Few people try to do great things, and I think the set of people who might have done any particular great thing is often quite small.
1Great comment Ajsja, you hit the mark. Two small comments:
(i) The 'correct', 'mathematically righteous' way to calculate credit is through a elaboration of counterfactual impact: the Shapley value. I believe it captures the things you want from credit that you write here.
(ii) On Turing being on the scent of information theory - I find this quote not that compelling. The idea of information as a logarithmic quantity was important but only a fraction of what Shannon did. In general, I agree with Schmidthuber's assesment that Turing' scientific stature is a little overrated.
A better comparison would probably be Ralph Hartley pioneered information-theoretic ideas (see e.g. the Shannon-Hartley theorem). I'm sure you know more about the history here than I do.
I'm certain one could write an entire book about the depth, significance and subtlety of Claude Shannon's work. Perennially underrated.
2Shapley seems like quite an arbitrary choice (why uniform over all coalitions?).
I think the actually mathematically right thing is just EDT/UDT, though this doesn't imply a clear notion of credit. (Maximizing shapley yields crazy results.)
Unfortunately, I don't think there is a correct notion of credit.
1Averaging over all coalitions seems quite natural to me; it averages out the "incidental, contigent, unfair" factor of who got in what coalition first. But tastes may differ.
Shapley value has many other good properties nailing it down as a canonical way to allocate credit.
Quoting from nunoSempere's article:
The Shapley value is uniquely determined by simple properties.
These properties:
- Property 1: Sum of the values adds up to the total value (Efficiency)
- Property 2: Equal agents have equal value (Symmetry)
- Property 3: Order indifference: it doesn't matter which order you go in (Linearity). Or, in other words, if there are two steps, Value(Step1 + Step2) = Value(Step1) + Value(Step2).
And an extra property:
- Property 4: Null-player (if in every world, adding a person to the world has no impact, the person has no impact). You can either take this as an axiom, or derive it from the first three properties.
In the context of scientific contributions, one might argue that property 1 & 2 are very natural, axiomatic while property 3 is merely very reasonable.
I agree Shapley value per se isn't the answer to all questions of credit. For instance, the Shapley value is not compositional: merging players into a single player doesn't preserve Shapley values.
Nevertheless, I feel it is a very good idea that has many or all properties people want when they talk about a right notion of credit.
- I don't know what you mean by UDT/EDT in this context - I would be super curious if you could elucidate! :)
- What do you mean by maximizing Shapley value gives crazy results? (as I point out above, Shapley value isn't the be all and end all of all questions of credit and in e.g. hierarchichal composition of agency isn't well-behaved).
1… and yet, Newton and Leibniz are among the most famous, highest-status mathematicians in history. In Leibniz’ case, he’s known almost exclusively for the invention of calculus. So even though Leibniz’ work was very clearly “just leading the parade”, our society has still assigned him very high status for leading that parade.
If I am to live in a society that assigns status at all, I would like it to assign status to people who try to solve the hard and important problems that aren't obviously going to be solved otherwise. I want people to, on the margin, do the things that seems like it wouldn't get done otherwise. But it seems plausible that sometimes, when someone tries to do this - and I do mean really tries to do this, and actually put a heroic effort toward solving something new and important, and actually succeeds...
... someone else solves it too, because you weren't working on something that hard to identify, even if it was very hard to identify in any other sense of the word. Reality doesn't seem that chaotic and humans that diverse for this to not have been the case a few times (though I don't have any examples here).
I wouldn't want those people to not have high status in this world where we're trying at all to assign high status to things for the right reasons. I think they probably chose the right things to work on, and the fact that there were other people who did as well through no way they could have easily known shouldn't count against them. Would Shannon's methodology have been any less meaningful if there were more highly intelligent people with the right mindset in the 20th century? What I want to reward is the approach, the mindset, the actually best reasonable effort to identify counterfactual impact, not the noisier signal. The opposite side of this incentive mechanism is people optimizing too hard for novelty where impact was more obvious. I don't know if Newton and Leibniz are in this category or not, but I sure feel uncertain about it.
I agree with everything else in this post very strongly, thank you for writing it.
The obvious but mediocre way to assign credit in the Newton-Leibniz case would be to give them each half credit.
The better but complicated version of that would be to estimate how many people would have counterfactually figured out calculus (within some timeframe) and then divide credit by that number. Which would still potentially give a fair bit of credit to Newton and Leibniz, since calculus was very impactful.
That sounds sensible, though admittedly complicated.
This is obviously the correct way to interpret what's happening. At some point the per person Shapley value becomes small but I'd guess that the shapely impact of Newton & Leibniz is substantial for quite a long time.
It's been a while since I read about this, but I think your slavery example might be a bit misleading. If I'm not mistaken, the movement to abolish slavery initially only gained serious steam in the United Kingdom. Adam Hochschild tells a story in Bury the Chains that makes the abolition of slavery look extremely contingent on the role activists played in shaping the UK political climate. A big piece of this story is how the UK used their might as a global superpower to help force an end to the transatlantic slave trade (as well as precedent setting).
I think you are completely overlooking a significant chunk of impact. Suppose that technologies A and B are similar. The techs act as substitutes, say several different designs of engine or something. And if everyone is using tech X, the accumulated experience makes X the better choice. This gives long term control of which path tech goes down to a "who got there first". Could electric cars have taken off before petrol if someone else had led that parade.
There are plenty of substances that increase fuel octane, so if someone else had led the parade around a substance that didn't contain lead, a lot of brain damage could have been prevented.
If some non military group had lead nuclear energy, would reactors use thorium instead of uranium?
Yeah, that sure does seem like a way it should be possible to have a lot of counterfactual impact. I'd be curious for any historical examples of people doing that both successfully and intentionally.
In Leibniz’ case, he’s known almost exclusively for the invention of calculus.
Was this supposed to be a joke (if so, consider me well and truly whooshed)? At any rate, it is most certainly not the case. Leibniz is known for a great many things (both within and without mathematics) as can be seen from a cursory glance at his Wikipedia page.
One measure of status is how far outside the field of accomplishment it extends. Using American public education as the standard, Leibniz is only known for calculus.
I am thinking of this as a noise-reducing modification to the loss function, similar to using model-based rather than model-free learning (which, if done well, rewards/punishes a policy based on the average reward/punishment it would have gotten over many steps).
If science were incentivized via prediction market (and assuming scientists can make sizable bets by taking out loans), then the first person to predict a thing wins most of the money related to it. In other words, prediction markets are approximately parade-leader-incentivizing.
But if there's a race to be the first to bet, then this reward is high-variance; Newton could get priority over Leibniz by getting his ideas to the market a little faster.
You recommend dividing credit more to all the people who could have gotten information to the market, with some kind of time-discount for when they could have done it. If we conceive of "who won the race" as introducing some noise into the credit-assignment, this is a way to de-noise things.
This has the consequence of taking away a lot of credit from race-winners when the race was pretty big, which is the part you focus on; based on this idea, you want to be part of smaller races (ideally size 1). But, outside-view, you should have wanted this all along anyway; if you are racing for status, but you are part of a big race, only a small number of people can win anyway, so your outside-view probability of personally winning status should already be divided by the number of racers. To think you have a good chance of winning such a race you must have personal reasons, and (since being in the race selects, in part, for people who think they can win) they're probably overconfident.
So for the most part your advice has no benefit for calibrated people, since being a parade-leader is hard.
There are for sure cases where your metric comes apart from expected-parade-leading by a lot more, though. A few years ago I heard accusations that one of the big names behind Deep Learning earned their status by visiting lots of research groups and keeping an eye out for what big things were going to happen next, and managing to publish papers on these big things just a bit ahead of everyone else. This strategy creates the appearance of being a fountain of information, when in fact the service provided is just a small speed boost to pre-existing trends. (I do not recall who exactly was being accused, and I don't have a lot of info on the reliability of this assessment anyway, it was just a rumor.)
Here's an example use-case where it matters a lot.
Suppose I'm trying to do a typical Progress Studies thing; I look at a bunch of historical examples of major discoveries and inventions, in hopes of finding useful patterns to emulate. For purposes of my data-gathering, I want to do the sort of credit assignment the post talks about: I want to filter out the parade-leaders, because I expect they're mostly dominated by noise.
I do think you're mostly-right for purposes of a researcher rationally optimizing for their own status. But that's a quite different use-case from an external observer trying to understand what factors drive progress, or even a researcher trying to understand key factors in order to boost their own work.
A sports analogy is Moneyball.
The counterfactual impact of a researcher is analogous to the insight that professional baseball players are largely interchangeable because they are all already selected from the extreme tail of baseball playing ability, which is to say the counterfactual impact of a given player added to the team is also low.
Of course in Moneyball they used this to get good-enough talent within budget, which is not the same as the researcher case. All of fantasy sports is exactly a giant counterfactual exercise; I wonder how far we could get with 'fantasy labs' or something.
One way to identify counterfactually-excellent researchers would be to compare the magnitude of their "greatest achievement" and secondary discoveries, because the credit that parade leaders get is often useful for propagating their future success and the people who do more with that boost are the ones who should be given extra credit for originality (their idea) as opposed to novelty (their idea first). Newton and Leibniz both had remarkably successful and diverse achievements, which suggests that they were relatively high in counterfactual impact in most (if not all) of those fields. Another approach would consider how many people or approaches to a problem had tried and failed to solve it: crediting the zeitgeist rather than Newton and/or Leibniz specifically seems to miss a critical question, namely that if neither of them solved it, would it have taken an additional year, or more like 10 to 50? In their case, we have a proxy to an answer: ideas took months or years to spread at all beyond the "centers of discovery" at the time, and so although they clearly took only a few months or years to compete for the prize of first (and a few decades to argue over it), we can relatively safely conjecture that whichever anonymous contender is third in the running is likely to have been behind on at least that timescale. That should be considered in contrast to Andrew Wiles, whose proof of Rermat's Last Theorem was efficiently and immediately published (and patched as needed) This is also important because other and in particular later luminaries of the field (e.g. Mengoli, Mercator, various Bernoullis, Euler, etc.) might not have had the vocabulary necessary to make as many discoveries as quickly as they did or communicate those discoveries as effectively if not for Newton & Leibniz's timely contributions.
I would say this is the kind of unconventional angle that I was originally hoping to see pumped out of leading AI alignment researchers when they take a break to look at AI policy instead. I'm not sure how much more can be gained from first principles and publicly available information, as opposed to spending time in DC and talking with people like Akash to get into the nitty gritty of the situation on the ground.
This reminds me of a pretty serious issue with history; beyond lists of events and dates of when they happened, any further detail gets sucked into disputes as people develop their own theories and get territorial when someone challenges them, and unlike the natural sciences, these hypotheses cannot really be tested since empiricism is dependent on historical documents (and, in recent history e.g. WW2 and Cold War, testimony from retired government officials and intelligence agency leaks, which gradually became around as unreliable as news reporting). Worst of all, incentives to forge written records pointing towards a giant revelation, that will get them lots of citations.
People who went to school will remember that, at the most nuanced parts of history class/textbooks, they would just give lists of causes for major events e.g. 4 major causes of WW1, 6 major causes of the corruption and collapse of Rome or a major Chinese Dynasty, and no weight placed on how much each of those factors contributed, just more lists of reasons how and why (e.g. was corruption one of the reasons that was 40% responsible or was it one of the 5% ones?).
This is why books like Guns, Germs, and Steel became so popular; it was just way above par, not adequate (and it also had to be easy to understand for a wider audience in order to get enough sales/citations to be famous, balancing that with actually being good in reality). There's no reference to bell curves or bayesian reasoning, and you can never even tell whether the scholar thinks humans are neurologically uniform, aside from Benjamin Franklin or Carnegie or Sun Tzu who were just badass™.
Afaik personal recommendations are required, but then you have to trust the people doing the recommending; will they select things maximally similar to Inadequate Equilibira e.g. only reading economic history because those are the only writers who sometimes bother to even attempt to think original thoughts about incentive structures?
Every time I think of doing research in a field that's too crowded, this post puts a faint image in my head of an obnoxious guy banging a drum at the front of a marching band. This is a real issue we need to keep in mind, both in AI safety research and elsewhere, and the number of LW posts that I remember at all two years later is pretty small, so this obviously deserves at least +1 review point.
The section on status made me pay more attention to my desire for status itself, but that's probably just me.
re: accumulating status in hope of future counterfactual impact.
I model status-qua-status (as opposed to status as a side effect of something real) as something like a score for "how good are you at cooperating with this particular machine?". The more you demonstrate cooperation, the more the machine will trust you. But you can't leverage that into getting the machine to do something different- that would immediately zero out your status/cooperation score.
There are exceptions. If you're exceptionally strategic you might make good use of that status by e.g. changing what the machine thinks it wants, or coopting the resources and splintering. But this isn't the vibe I get from people I talk to with the 'status then impact' plan, or from any of 80ks advice. They sound like they think status is a fungible resource that can be spent anywhere, like money[1].
So unless you start with a goal and authentically backchain into a plan where a set amount of a specific form of status is a key resource, you probably shouldn't accumulate status.
I think money-then-impact plans risk being nonterminating, but are great if they are responsive and will terminate.
I also think getting a few years of normal work under your belt between college and crazy independent work can be a real asset, as long as you avoid the just-one-more-year trap.
People occasionally come up with plans like "I'll lead the parade for a while, thereby accumulating high status. Then, I'll use that high status to counterfactually influence things!". This is one subcategory of a more general class of plans: "I'll chase status for a while, then use that status to counterfactually influence things!". Various versions of this often come from EAs who are planning to get a machine learning PhD or work at one of the big three AI labs.
Even if you need status, it might be easier to just be or become friends with the people who already have status and credentials, and they can lend you their status if they think your plan/idea is good. For example, when you are writing a letter to a politician or founding a new org/startup.
I like the idea of this leading the parade analogy, but I feel there are two concepts here, leading the parade in terms of a powerless figurehead, and a distinct idea. One I more science, the other more politics.
Whether or not your research has counterfactual impact depends not just on what you do, but also on what everyone else in the world does. If there are 100 problems to work on, ranked in "interesting-ness" from top to bottom and you are one of 100 researchers, how do you maximise your counterfactual impact? Obviously not by working on the biggest problems, the "most interesting" problem will probably have 5 other people doing it. (The game-theory thing where "If we all go for the blonde" https://plus.maths.org/content/if-we-all-go-blonde ).
The theory does tell us that if we happen to know problem X is way more interesting than most people give it credit for we should prioritise that, but that's not much of an insight.
I think "leading the parade" has a much more interesting application to politics. Imagine the leader of a political party - depending on the leader, and the party, the party leader might have genuine, power to change the party's policies or political aims, but (usually?) the party already has its "thing" and the leader can only adjust within that. Donald Trump is an example of someone who clearly has real power to change the sorts of things the Republicans stand for, because he already has. In contrast, most party leaders just do what that party would do anyway. That seems like a really important thing to know. Imagine the prime minister has some policy that you really don't like, you manage to get a meeting with them and explain why the policy is bad. Then, in a weird moment of perfect honesty, they look you in the eye and say "yeah, on a personal level I completely agree this policy is bad. But my party are behind this, and its happening with or without my backing. Why are you even talking to me, I am just the prime minister, I lead this party but I don't control it."
(Unrelated, but a few years ago I published a paper that I really thought was like, some crazy thing that would only be seen by someone with my weird perspective on the issue - proper counterfactual impact. It only took 2 years for someone else to independently come up with the same idea (and the same figures!) and put up a preprint of their own.)
1William Thurston seems like a mathematician that was not just leading the parade, but rather made fundamental contributions to his fields that no one else would have made in his time. To quote, "He wanted to avoid in hyperbolic geometry what had happened when his basic papers on foliations “tsunamied” the field in the early 1970s.", meaning that he made so many deep and varied contributions so fast that no one could keep up. More in https://www.ams.org/notices/201511/rnoti-p1318.pdf such as "The huge and daunting advances he made in foliation theory were off-putting, and students stopped going into the area, resulting in an unfortunate premature arrest in the development of the subject while it was still in its prime. (If someone writes a book incorporating Bill’s advances, it will take off again.)"
1I feel like it's gotta be pretty tricky to fully eliminate the possibility that someone had counterfactual impact when there was simultaneous invention, if the person who had counterfactual impact had been talking about their partially developed ideas beforehand. It could lead to others developing them further.
People occasionally come up with plans like "I'll lead the parade for a while, thereby accumulating high status. Then, I'll use that high status to counterfactually influence things!". This is one subcategory of a more general class of plans: "I'll chase status for a while, then use that status to counterfactually influence things!". Various versions of this often come from EAs who are planning to get a machine learning PhD or work at one of the big three AI labs.
This but skill instead of status?
The people who I know who currently seem most impressive spent a lot of time earlier on gaining a bunch of skill. I don't know anyone who seems impressive by virtue of having gained lots of status which they are now free to divert to good ends. Perhaps I just don't hang around such people, but for this reason I'm much less convinced of John's arguments when you replace status with skill.
Sorry, yeah, my comment was quite ambiguous.
I meant that while gaining status might be a questionable first step in a plan to have impact, gaining skill is pretty much an essential one, and in particular getting an ML PhD or working at a big lab seem like quite solid plans for gaining skill.
i.e. if you replace status with skill I agree with the quotes instead of John.
2+1, "gain a bunch of skill and then use it to counterfactually influence things" seems very sensible. If the plan is to gain a bunch of skill by leading a parade, then I'm somewhat more skeptical about whether that's really the best strategy for skill gain, but I could imagine situations where that's plausible.
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
Curated and popular this week
