I disagree with the author in that I believe the universe is 'compact' in the sense that what we know is a near approximation of what is knowable by the method. 

For example, I believe our ancestors knew everything they can possibly know without knowing extra background knowledge or special kinds of thinking. 

Similarly, I believe we now know nearly everything about the universe (the rules, not specific tools) given use of mathematics and sophisticated logic. And we do a much, much better job than our ancestors at predicting the world. So I am skeptical that superintelligence can exist.

You may say, how can you assume whatever we have today is the final stages of logic? Why can't this be the very beginning? Again, we nearly explained the universe by now. Second, we have so many interconnected people that it is more than any community in old times by orders of magnitude. So even though logic is harder, I think it is mostly explored.

But this wouldn't negate AGI, for AGI is different from superintelligence in that we know it is possible (humans, sort of), and also much less powerful than superintelligence. 

But one thing we must remember is the hollow nature of economic size. Economy is basically about extracting resources to satisfy our needs. Currently, large economics size is meaningful because it means more resources for each human. And even if a flood of AGI agents increases resource extraction, sums of money in accounts of mere sophisticated algorithms wouldn't mean anything. At least, not to me.

What I understood: We wouldn't be automatically right by saying the opposite of what stupid people say. This is because stupid people are wrong from many angles, and to be right one needs to be right in all angles. And some times, stupid people may be correct too. Or rather, stupidity of believers is not evidence against the belief. In fact, most beliefs are not completely wrong. We want to keep the correct parts and reject wrong parts, not reject everything once we realize it fails to predict our goals.

I understand that you are saying that even a single resource lacking for one to "thrive" is poverty, that poor people are not thriving today because they lack some resource that they need to thrive. This resource was not provided to them even after the society became much more productive, and hence probably would not be provided to them if we implement UBI. You try to show this using a counter factual country where the critical resource is oxygen.

I think this is a false equivalence. The 'equilibrium' that enforces poverty is actually people themselves. I think the resource lacking is not something essential to simply survive. Simple survival, after all, can be taken care of at a minimal cost if people can just move to some minimal remote place and raise chickens to eat or something. But they don't, because they have needs that take money to fulfill. These 'needs' are not present before, but are present now, simply because it takes a lot more now for one to be happy. 

Curiously, there is a theory about how schizophrenia is due to differences in dimentionality of neurons in different parts of the brain. I wonder what you think about it?

Thankyou for pointing out holes in my argument. 

I don't think Google search engine is an entity that I call a demon of statistics. 

I classify thought processes as algorithmic and statistical. The former merely depends on IQ, while the later is more subjective,  based on mental models. I am thinking along lines parallel to JonahS in his posts on mathematical ability.

To explain my reasoning, I think while it is difficult to distinguish simple statistical machines (as in smart keyboards, search engines) differ from demons of statistics, we must distinguish them based on their position in intelligence space.

Search engines do not give you sentences, but the result associated with the query, as I understand it. This may use statistical methods, but it does not overlap with statistical thinking of humans in intelligence space.

On the other hand, LLMs do overlap with human intelligence space, in their statistical thinking aspect.

I think depending on machines that overlap with statistical aspect (and higher levels) of human intelligence is where one starts to lose humanity. I don't distinguish between 'post human' and Inhuman.

On the other hand, algorithmic machines are age old, and using simple ones like beads for counting does not deprive one of humanity.

Also, regarding books, I think there is no difference between consulting them and asking your grandma (or any person much older than you), since I accept algorithmic machines.

No, I think the ship never changed. As long as the structure is same, parts do not matter. This is the virtue of statistical thinking, and the same as how you recognize a dog when you see it.

Finally, I agree that we can never reach true post human, only become less human. One exception is if everyone commits suicide as described in this post. I think this is even more dangerous than bad AI, since AI can be stopped, but humanity cannot be interfered with, given our morality.

As a materialist, I disagree so early with your chain of thought that we share only a little of our worldview. Our disagreement started when you start with

The Veil of Ignorance, but adequate.

This though experiment is interesting to read, but delves too far from reality. I find that it makes it very easy to mistake the map for territory.

But trivialities aside, I see that the thought experiment tries to construct the idea of a society that the thinker finds to be good enough, on average. But this is inherently flawed, since there are too many unknowns to even start.

Then comes the 

you never had the chance of choosing who you are born as

point. This cannot happen, as 'you' can come into being only after birth! 

Then, the consciousness nonsense. Consciousness is a physical process, not a metaphysical one. Only an entity similar 'enough' to humans can be conscious.

Algorithms affirming themselves is not happiness, for there is no consciousness. Only when one realizes that they are nothing more then a body does everything become clear.

I disagree with your first point. You are saying people who use a tool are already 'post human' in some sense. But then, are people who can use abacus in 14th century post human? Are African tribes that use their technical knowledge to hunt animals, less human than a hypothetical tribe that never got to use anything like a spear, and fight with their bare hands? By that logic, chimps are more 'human' than humans!

I think we can draw a line. Algorithms are more or less things tools that give answers to what we want. It is a mistake to think they are above humans; computers just let us effectively use them. Is a person using LLMs in work human? No to me. But purely algorithmic tools get a pass. The point is that when AIs inform us, they take away a part of our 'agency'.

One might ask, how is this different from asking answer from just another person? My answer is, the one they asked is not a human. It is a 'demon of statistics', as I call it. It is something that knows statistical associations between every word on the internet, and can construct meaning based on this alone. This is clearly beyond human capability. Note that my distinction is based on my belief that knowledge of a 'demon of statistics' is fundamentally different from that of humans.

But take the part of the story where the protagonist stops thinking with his brain, and gives up decision making to AI that is 'similar' to him. This is not human, and it is where I draw the line. But with his usage of AIs from the start, we can also argue he was never fully 'human' to begin with. We who were born before the birth of AI can be considered the last of 'true' humanity.

Using this definition, anyone who uses, say, Chat GPT, to make any decision for them, even for a small part of their life, are already no longer human. But they can revert to a human again if the decisions informed by AI no longer effect them, something potentially effectively impossible in an AI dominated world.

I consulted Chat GPT about this very paragraph now, and it replied if I considered someone consults AI, and makes the final decision themselves, are they still human?

And my reply was this:

What does 'making decision themselves' even mean? If one can consider the wisdom of an entity that can form meaningful sentences distilled by whole knowledge of humanity, even as a part of their decision, how can they be still called human? But they are a 'non human' with respect to that decision. Even in their life, their non-humanness can be considered to have been increased.

So, I am already not just a human, a different being. That said, I am all in for the Butlerarian Jihad... If humans can do the work, let no AI do it!

I restate my comment on Substack here.

I think this is the story of an AI, not a human. This is a future I find horrifying, where humanity dies out, never realizing it until the end. Many here seem to think it is enough as long as a super intelligence does not wipe out humanity, helping it instead. But for humanity, any being that makes humanity redundant is a death knell in the long run. This is a kind of Moloch situation.

To go into the specifics, when the author seem to use the 'Ship of Theseus' argument, he did not seem to realize that if the boat is dismantled piece by piece and a house is built with the pieces, it is definitely no longer a ship, let alone the same ship. In fact, the change starts with internalization of AI, or more precisely, when the protagonist stopped using his biological mind in favor of AI making the decisions. 

I do not think 'growth mindset' is necessary for growth if one understands what 'talent' really means. I define 'talent' in a task as competence at learning new things in that particular task. I think people generally see their current learning speed as limited by their 'talent', but it is actually limited by concentration/effort/dedication. 

After certain stage, the later matter a lot more than what many think. We also see that people with growth mindset do not improve their talent, but the other things. It would be good if people with fixed mindset realize that talent is not everything. This is not a question of 'mindset', but unbiased review of competence function.

An excellent reply to Ms. McGonall. Just what I would expect of Harry potter with a brain. I have to reaffirm, this is the best Harry Potter fan fiction out there!