LessWrong's voting system might bury content which would otherwise make rationalists aware of inconsistencies, but it may also bury content which would otherwise convince rationalists to disregard flagged inconsistencies. I suspect that the voting system does more good than bad for group epistemics, but I think evidence is necessary to defend strong claims for either position.
Every group of people will have some features in common with the prototypical cult. I don't think it's useful to refer to rationalism as a cult because I doubt that it has enough cultish features. For example: there is no authoritarian leader, restrictions are not imposed on rationalists' contact with family and friends, etc.
My understanding of your post: If an ASI predicts that in the future it's goal will change to X, the agent will start pursuing X instead of the goal it was given at initialisation, Y. Even if we figured out how to set Y correctly, that would not be sufficient. We would also have to ensure that the agents goal could never change to X, and this is not possible.
I have a few misgivings about this argument, most significantly:
Why does the agent care about pursuing X? Maybe it cares about how successful its future self is, but why? If we ablate some parameters of your example, I think it pumps the intuition that the agent does not care about X. For example:
- Suppose that building the paperclip factory reduces the expected number of cups by a tiny amount. In this case, the agent doesn't build the factory until it's goal is changed to X.
- Suppose the agent has more than one action available to them. If any action increases the expected number of cups by even a tiny amount, the agent takes this action instead of building the paperclip factory.
- If the agent has an action which reduces the likelihood of its goal being changed to X, the agent takes this action because it increases the expected number of cups.
- If the agent is not able to predict what it's new goal will be, it does not build the paperclip factory. The new goal could just as easily be to minimise the number of paperclips as it could be to maximise the number of paperclips.