Krantz

Much in the way that we have prosthetic limbs 'do' what we think they should do, we will soon have rational agents that link to fully embodied machines that 'listen to what we think' and simply 'do' what we think they ought. Let me give you an example of this. Let's say we get to the point where I can train a personal GPT to have the exact same epistemology as me. Feed it a tremendous amount of data about me. An then give it access to listen in on my stream of consciousness. It could act on my behalf. I simply think 'man, I should write up this idea and post it on Lesswrong' and my GPT facsimile listening thinks 'Well, since he has given me permission to do this in the past AND I think he would want me to go ahead and do that for him, I will.'. This could be very dangerous if we extrapolate this to fleets of machine drones. Obviously. You effectively are the machines you have. This presents a different future. Not a future where there is a conflict between 'the goals of humans' and 'the goals of AI'. It's a future where we become the machines. We become the central nervous system for the brain that is 'collective ASI'. The goal of the brain is to listen to the central nervous system. Alignment is the field of ensuring the brain represents the CNS in a fair and balanced way. Collective intelligence (knowledge mapping in general) is the field of formally demonstrating the values and perspectives of agents within a large group. It's about gathering user data that will be valuable to an altruistic intelligence about what people would actually like to happen. This is the whole reason government was created. Cryptography and decentralization play a critical role in ensuring a fair and balanced operation of the consensus gathering mechanisms. Major work is being done using collective intelligence consensus methods for establishing constitutions for AI to be aligned to. Prosthetic intelligence also has its benefits. In the case we achieve decentralized