Given that the process of scientific research has many AGI traits (opaque, self-improving, amoral as a whole), I wonder how rational it is for laypersons to trust it. I suspect the answer is, not very. Primarily because, just like an AGI improving itself, it doesn't seem to be possible for anyone, not even insiders in the process, to actually guarantee the process will not, in its endless iterations, produce an X-risk. And indeed, said process is the only plausible source of manmade X-risk. This is basically Bostrom's technological black ball thought experiment in the Vulnerable World Hypothesis. But Bostrom's proposed solution is to double down, with his panopticon.
I have an intuition that such instances of doubling down are indications the scientific research process itself is misaligned.
Alignment/misalignment makes more sense for something agentic, pursuing a goal. To be safe, an agent must be aligned, but another way of making a possibly-misaligned possibly-agent safer involves making it less agentic. A dangerous non-agent is not misaligned, its danger must be understood in some other way that would actually explain what's going on.
The thing is though, there isn't a dichotomy between agents and processes. Everything physical (except maybe the elementary particles) is a process in the final analysis, as Heraclitus claimed. Even actual individual persons, the paradigmatic examples of an agent, are also processes: the activity of the brain and body only ever stop at death. The appearance of people as monadic agents is just that, an appearance, and not actually real.
This might have sounded too much philosophical woo-woo, but it does have pragmatic consequences, which is that since agents... (read more)