In the previous post, I talked about several "anchors" that we could use to think about future ML systems, including current ML systems, humans, ideal optimizers, and complex systems.

In fact, I think we should be using all of these anchors (and any others we can think of) to reason about future ML systems. This is based on ideas from forecasting, where successful forecasters usually average over many worldviews and reference classes rather than focusing on a single reference class. However, we should also be discerning and weight anchors more if they seem like a better match for what we want to predict.

Below, I'll say what I personally think about most of the anchors we discussed so far, by assigning a numerical "weight" to each one. While these weights aren't perfect (the actual weight I'd use depends on the particular question), they hopefully provide a clear overall picture that is easy to agree/disagree with.

Here are the rough weights I came up with:

I primarily rely on Current ML, Complex Systems, and Thought Experiments, in a 4:3:2 ratio. In particular, I assign about twice as much weight to Current ML as to Thought Experiments, but I think the opposite ratio is also defensible. However, many people seem to implicitly put almost all their weight on Current ML, or almost all their weight on Thought Experiments. They have something like a 5:1 or 1:5 ratio, or even greater. I think neither of these stances is defensible, and I would be interested in anyone who disagrees writing up the case for assigning extreme weights (in either direction).

Relatedly, my last two posts were essentially an argument against a 5:1 ratio in favor of Current ML--first by arguing that Current ML often misses important developments, and second by arguing that thought experiments can sometimes catch these.^[1]

Aside from this, my biggest disagreement with others would be assigning significant weight to the "Complex Systems" anchor, which I think most people overlook.

Finally, all anchors that correspond to a broad reference class (Current ML, Complex Systems, Thought Experiments) get significantly more weight than any anchor that is a single example (e.g. humans).^[2] I give serious consideration to hypotheses generated by any of these three anchors. In particular, if I can't strongly rule out the hypothesis after one hour of thought, I think there's at least a 30% chance that it will eventually come to be supported by the other two anchors as well.

I'd be interested in others posting their relative weights, and pointing out any instances where they think I'm wrong.