I think tool AIs are very dangerous because they allow the application of undigested knowledge
Another form of this is asking questions that are motivated by obtaining material goods for the self. More altruistic questions that seek first the profit of others seem less privileged in general.
It is better for one's health and the planet in terms of emissionsto eat less meat as well.
A breakdown of the risks
It sounds like mathematical platonism which appeals to some like (seemingly) Roger Penrose, but it seems connected to other networks of concepts at least in part and I do not think it should be taken as a given. Perhaps in the future we can model when such belief systems will arise given access to other information based on the history of the individual in questiin, but we are not quite there yet. For further reference one could examine the examples of computationally model led social behavior in a topic someone created here regarding the reverse engineering of belief systems
Using the non - cognitive approach you could dismiss statements in symbolic logic that do not refer to constructs or events that could come into being. I am referring to constructs in Deutsch's theory
I wish the audio was available for free
Brains are like cars. Some are trucks made for heavy hauling, but travel slow. Some are sedans: economical, fuel efficient, but not very exciting. Some are ferraris sleek, fast, and sexy but burn through resources like a mother. I'm sure you can come up with more of your own analogies.