I've talked with and interviewed a lot of software developers, and it's probably fewer than 5% that really understand QuickSort including the variance in performance on pathological lists. This is trivially simple compared to large models, but not actually easy or self-explaining.

Well, these programmers probably didn't try to understand Quicksort. I think you can see simple dynamics such as, "oh this will always return a list that is the same size as the list that I input" and "all the elements in that list will be elements from the original list in a bijective mapping. There won't be different elements and there won't be duplicated elements or something like that." That part is pretty easy to see. And now there are some pathological cases for quick search, though I don't understand the mechanics of why they arise. However, I'm pretty sure that I can, within one hour, understand very well what these pathological cases are and why they arise, and how I might change a quick search algorithm to handle a particular pathological case well. That is, I'm not saying I look at Wikipedia and just read up on the pathological cases, but I just look at the algorithm alone and then derive the pathological cases. Maybe an hour is not enough, I'm not sure. That seems like an interesting experiment to test my claim.

I am pretty sure that there is a program that you can write down that has the same structural property of being interpretable in this way, where the algorithm also happens to define an AGI.

I am pretty sure that this is not possible.

Could you explain why you think that this is not possible? Do you really think there isn't an explicit Python program that I can write down such that within the Python program, e.g. write down the step-by-step instructions that when you follow them, you will end up building an accurate model of the world. And such that the program does not use any layered optimization like SGD or something similar. Do you think these kinds of instructions don't exist? Well, if they don't exist, how does the neural network learn things like constructing a world model? How does the human brain do it?

Once you write down your algorithm explicitly like that, I just expect that it will have this structural property I'm talking about of being possible to analyze and get intuitions about the algorithm.

That is an interesting analogy.

So if I have a simple AGI algorithm, then if I can predict where it will move to, and understand the final state it will move to, I am probably good, as long as I can be sure of some high-level properties of the plan. I.e. the plan should not take over the world let's say. That seems to be property you might be able to predict of a plan, because it would make the plan so much longer, than just doing the obvious thing. This isn't easy of cause, but I don't think having a system that is more complex would help with this. Having a system that is simple makes it simpler to analyze the system in all regards, all else equal (assuming you don't make it short by writing a code golf program, you still want to follow good design practices, and lay out the program in the obvious most understandable way).

As a story sidenote before I get into why I think tho Q-gap probably is wrong: That I can't predict that it will rain tomorrow if I have the perfect model of low-level dynamics in the universe, has more to do with how much compute I have available. I might be able to predict if it would rain tomorrow would I know the initial conditions of the universe and some very large but finite amount of compute, if the universe is not infinite?

I am not sure the Q-gap makes sense. I can have a 2D double pendulum. This is very easy to describe and hard to predict. I can make a chaotic system more complex, and then it becomes a bit harder to predict but not really by much. It's not analytically solvable for 2 joints already (according to Google).

That describing the functioning of complex mechanisms seems harder than saying what they do, might be an illusion. We as humans have a lot of abstractions in our heads to think about the real world. A lot of the things that we build mechanisms to do are expressible in these concepts. So they seem simple to us. This is true for most mechanisms we build that produce some observable output.

If we ask "What does this game program running of a computer do?" We can say something like "It creates the world that I see on the screen." That is a simple explanation in terms of observed effects. We care about things in the world, and for those things we normally have concepts, and then machines that manipulate the world in ways we want have interpretable output.

There is also the factor that we need complex programs for things where we have not figured out a good general solution, which would then be simple. If we have a complex program in the world, it might be complex because the creators have not figured out how to do it the right way.

So I guess I am saying that there are two properties of a program. Caoticness, and Kolmogorov complexity. Increasing one always makes the program less interpretable, if the other stays fixed, if we assume that we are only considering optimal algorithms, and not a bunch of halfhazard heuristics we use because we have not figured out the best algorithm yet.

I expect that it is much more likely that most people are looking at the current state of the art and don't even know or think about other possible systems and just narrowly focus on aligning the state of the art, not considering creating a "new paradigm", because they think that would just take too long.

I would be surprised if there were a lot of people who carefully thought about the topic and used the following reasoning procedure:

"Well, we could build AGI in an understandable way, where we just discover the algorithms of intelligence. But this would be bad because then we would understand intelligence very well, which means that the system is very capable. So because we understand it so well now, it makes it easier for us to figure out how to do lots of more capability stuff with the system, like making it recursively self-improving. Also, if the system is inherently more understandable, then it would also be easier for the AI to self-modify because understanding itself would be easier. So all of this seems bad, so instead we shouldn't try to understand our systems. Instead, we should use neural networks, which we don't understand at all, and use SGD in order to optimize the parameters of the neural network such that they correspond to the algorithms of intelligence, but are represented in such a format that we have no idea what's going on at all. That is much safer because now it will be harder to understand the algorithms of intelligence, making it harder to improve and use. Also if an AI would look at itself as a neural network, it would be at least a bit harder for it to figure out how to recursively self-improve."

Obviously, alignment is a really hard problem and it is actually very helpful to understand what is going on in your system at the algorithmic level in order to figure out what's wrong with that specific algorithm. How is it not aligned? And how would we need to change it in order to make it aligned? At least, that's what I expect. I think not using an approach where the system is interpretable hurts alignment more than capabilities. People have been steadily making progress at making our systems more capable and not understanding them at all, in terms of what algorithms they run inside, doesn't seem to be much of an issue there, however for alignment that's a huge issue.

I think the problem with the things you mention is that they are just super vague, where you don't even know what is the thing that you are talking about. What does it mean that:

Most important of all, perhaps, is making such machines learn from their own experience.

Finally, we'll get machines that think about themselves and make up theories, good or bad, of how they, themselves might work.

Also, all of this seems to be some sort of vague stuff about imagining how AI systems could be. I'm actually interested in just making the AI systems and making them in a very specific way such that they have good alignment properties and not vaguely philosophizing about what could happen. The whole point of writing down algorithms explicitly, which is one non-dumb way to build AGI, is that you can just see what's going on in the algorithm and understand it and design the algorithm in such a way that it would think in a very particular way.

So it's not like, oh yes, these machines will think for themselves some stuff and it will be good or bad, it's more like, I make these machines think, how do I make them think, what's the actual algorithm to make them think, how can I make this algorithm such that it will actually be aligned. And I am controlling what they are thinking, I am controlling if it's good or bad, I am controlling if they are going to build a model of themselves, maybe that's dangerous for alignment purposes in some context and then I would want the algorithm to not want the system to build a model of themselves.

For, at that point, they'll probably object to being called machines. I think it's pretty accurate to say that I am a machine.

(Also, as a meta note, it would be very good, I think, if you do not break the lines as you did in this big text block because that's pretty annoying to block quote.)