It's worth noting that we observe other forms of simplication of language as well. English reduced the amount of inflections of verbs. The distinction between singular and plural pronouns disappeared.
In many cases, there are diminishing returns to a given scientific paradigm. The fact that you observe a field getting diminishing returns doesn't mean that there isn't a paradigm that the field could adopt that would allow for returns to flow again. Paradigm change is about pursuing ideas that people in the old paradigm don't find promising.
Just adding more smart people who follow a hegemonic paradigm doesn't automatically get you paradigm shifts that unlock new returns. If string theory stiffles progress, it would look from the inside like there are diminishing returns to theoretical physics.
There seems to be papers that show that if you naively train on chain of thought, you train models not to verbalize potentially problematic reasoning in their chain of thought. I however don't see discussion about how to train chain of thought models to better verbalize their reasoning.
If you can easily train a model to hide it's reasoning you should also be able to train models the other way around to be more explicit about their reasoning.
One approach I imagine is to take a query like diagnosing medical issues and replace key words that change the output and then see how well the chain of thought reflects that change. If the chain of thought tells you something about the change in outcome, you reinforce the chain of thought. If the chain of thought doesn't reflect the outcome well, you punish the chain of thought.
All it takes is trusting that people believe what they say over and over for decades across all of society, and getting all your evidence about reality filtered through those same people.
I seems to me like you also need to have no desire to figure things out on your own. A lot of rationalists have experiences of seeking truth and finding out that certain beliefs people around them hold aren't true. Rationalists who grow up in communities where many people believe in God frequently deconvert because they see enough signs that the beliefs of those people around them aren't really fitting together.
Given that most people living in religious communities grow up believing in God just as the people around them do, it's might be very normal to think that way, but it still feels really strange to me and probably does feel strange to many other rationalists as well.
What do you mean with 'must'? The word has to different meanings in this context and it seems bad epistemology not to distinguish them.
Have you thought about making an altered version that strips out enough of the My Little Pony-IP to be able to sell the book on Amazon KDP? (or let someone else do that for you if you don't want to do the work?)
The existing ontology that we have around consciousness is pretty unclear. A better understanding the nature of consciousness and thus what's valuable will likely come with new ontology.
When it comes to reasoning around statistics, robustness of judgements, causality, what it means not to Goodhart it's likely that getting better at reasoning also means to come up with new ontology.
Regardless of the details, we ought to prioritize taking all of our power plants, water purification stations, and nuclear facilities out of the world-wide-web.
I think it's very questionable, to make major safety policy "regardless of the details". If you want to increase the safety of power plants, listening to the people who are responsible for the safety of power plants and their analysis of the details, is likely a better step instead of making these kind of decisions without understanding the details.
Orcas already seem to have language to communicate with other orcas. Before trying to teach them a new language, it would make more sense to better understand the capabilities of their existing language and maybe think about how it could be extended to communicate with them about what humans want to talk about with them.
I think the problem is that you ignore the idea that science works via paradigms. Even if there's a possible paradigm besides string theory that would produce more progress, there are a lot of different things that people who aren't working on string theory could work on. Most of them won't lead anywhere.
If a new paradigm could be found that has more potential, that paradigm would have new low hanging fruit.
However, researchers that would write papers about that low hanging fruit, might have trouble getting published in journals of the old paradigm because they are solving problems of interest to the new paradigm and not problems of interest of the old paradigm. Getting funding to work on problems of a new paradigm is also harder.