“The people you need to soften/moderate your message to reach (or who need social proof in order to get involved) are seldom going to be the ones who can think clearly about this stuff. And we are very bottlenecked on high-quality thinking.”.

 

I think this is true only in a part of contexts. If we are talking about AI alignment - probably skilled mathematicians or AI researches can be very fit. At least in the directions like interpretability. And this doesn’t necessarily correlate with their desires to do societally unconventional work. Why isn’t that so?

What about the enhancement of human intelligence that was discussed here? (For example How to make superkids - LessWrong).

They probably have more than a 1% chance of success and could accelerate anti-aging research. Even if you consider the current research situation critically stalled.  

and yet you do not identify any of these, supposed "other useful properties".  

I did it in the second paragraph.

How can you reconcile a prediction of algorithmic breakthroughs with reality? When would that reconciliation take place. Nobody is ever going to look back and say "I predicted algorithmic breakthroughs and there were none"

Maybe many people would do that, but I think at least some would be able to acnowledge the mistake and not rationalize away their prediction. To reconcile their prediction with reality, one, as an option, can make the prediction very concrete and narrow in the beginning. And that's what people here generally try to do, as I see it. 

As I understand it, the stated goal of the text was to build a concrete scenario despite this problem. Even though increased concreteness reduces the probability of a particular scenario from the cluster, it has other useful properties.

For example, it is probably easier and more interesting to analyze a concrete story. Building one and reconciling it with reality can be an exercise that improves one`s ability to make predictions.

I understand it much better now, thanks! (I did not know about Tencent, and I foolishly had not read that footnote carefully enough.)

Although I don't fully understand why drawing attention to two companies is viewed acceptable when drawing attention to only one company is not. As said, other companies also have a notable chance of becoming the leader.

Probably drawing attention to one particular company can be seen as a targeted attack or advertisement, and one is much less likely to advertise for/attack two companies at the same time. But one could be “playing for” other company (say, X AI or Anthropic) and thereby be attacking other leading companies?

(To avoid singling out any one existing company, we’re going to describe a fictional artificial general intelligence company, which we’ll call OpenBrain. We imagine the others to be 3–9 months behind OpenBrain.)

I am confused/curious. Doesn't "OpenBrain" closely resemble "OpenAI", thereby singling out an existing company? Similarly, "DeepCent" in this context closely resembles "DeepSeek".

Maybe this is humor, or there are legal reasons for not using real names?

To avoid singling out existing companies, it might be better to name the company "Wonderful AI", "Agentic AI", "AI company №352", etc. On the other side, using real names could be better for the goal of making the prediction more concrete and serious (the use of "OpenBrain" may evoke feelings of an alternative reality).

In such a case one should probably engage in independent research until they have developed the relevant skills well enough (and they know it). After that point, persisting in independent research rather than seeking help can be an unproductive use of time. Although it is not obvious how attainable this point is.