The ability to get to consciously decide when to discard or rewrite or call on the simple programs is a superpower evolution didn't give humans. One that it seems would be the obvious solution for an AI that gets to call on an external, updatable set of tools. Or an ASI got got to rewrite the parts of itself that call the tools or notice (what it previously thought were) edge cases.
AKA, an ASI can go ahead and have a human-specific prior. It can choose to apply it until it meets entities that are alien, then stop applying it. Humans can't really do that, in the same way that we can't turn off our visual heuristics when encountering things we consciously know are weirdly constructed adversarial examples, even if we can sometimes override them with enough effort. The ASI, presumably, would further react to encountering aliens by reasoning from more basic principles (recurse as needed) as it learns enough to create 1) a new prior specific to those aliens, 2) a new prior specific to those aliens' species, culture, world, etc.
Or at least, that's my <4 minute human-level single attempt at guessing a lower bound on an ASI's solution.
I think if you start having meta-priors, then what, you gotta have meta-meta-priors and so on? At some point that's just having more basic, fundamental priors that embrace a wider range of possibilities. The question is what would those look like, or if being general enough doesn't descend into a completely uniform (or very little informative) prior that is essentially of no help; you can think anything, but the trade-off is it's always going to be inefficient.
True, but I think in this case there's at least no risk of an infinite regress. At one end, yes, it bottoms out in an extremely vague and inefficient but general hyperprior. I would guess from the little I've read that in humans these are the layers that govern how we learn from even before we're born. I would imagine an ASI would have at least one layer more fundamental than this, which enable it to change various fixed-in-humans assumptions about things.
At the other end would be the most specific or most abstracted layer of priors that has proven useful to date. Somewhere in the stack are your current best processes for deciding whether particular priors or layers of priors are useful or worth keeping or if you need a new one.
I am actually not sure whether 'prior' is quite the right term here? Some of it feels like the distinction between thingspace and conceptspace, where the priors might be more about the expectations what things exist and where natural concept boundaries lie and how to evaluate and re-evaluate those?
Thinking about intelligence, I've decided to try something. The ARC-AGI puzzles are open and free for anyone to try via a convenient web interface. Having a decently well-trained general intelligence at hand in my own head, if I'm allowed the immodesty, I decided to make a little experiment: tackle one of them (the Daily Puzzle, to be precise), and record myself doing so. I'm going to talk about it and spoil it for you if you haven't played it yet, so if you want to try, go do so first.
The solution
Here are the screenshots for context:
First stage: analysis of the patterns.
Stage two, formulating a hypothesis.
(the above refers to example 1 by the way)
(the above refers to examples 2 and 3)
At this point the solution board looked like this:
Analysis
I finished the problem in 4 minutes and 9 seconds (at least 45 seconds of which were spent taking screenshots) and with two attempts. I could have probably gone faster had I not been speaking the whole time, and the attempts I feel like were fundamentally just a matter of luck. While I'm not a hardcore puzzle solver, I've got my fair experience with them (and I have solved quite a few ARC ones before) and I probably have way more of a "puzzle brain" than the average person, so I would guess this time would still be fairly good.
A few sparse observations:
The interesting conclusion to me is that two things here did almost all of the work. One of them is pattern recognition, a mechanism that seems completely unconscious and I imagine some sort of sophisticated elaboration on convolution and/or attention, but noticeably, heavily biased to detect similarity even past specific operations that are assumed more likely and yet not enough to break correspondence (like translation or filling). The second is a lot of priors, and these priors are very very specific. I don't think they're even necessarily about things being "simpler" in some absolute, Solomonoff sense. They just feel like they're about puzzles, and how puzzles, or code, or things designed for us, work. This probably speaks in part to the limits of the ARC dataset (though this is one of the easiest problem I've found from it; others are much less intuitive). But it also makes me think about the concept of AGI and ASI. Ideally, we imagine a superintelligence being able to spot much more sophisticated patterns than we do. This implies a certain untethering from our own biases and limits. And yet our ability to solve certain categories of problems in the first place seems to be dictated by how we have a ready-made library of concepts that we strongly select on when trying to navigate the tree of all possible explanations. As if what makes us a general intelligence is having a collection of simple programs that are factually very useful, and an algorithm well trained on building a tree out of them to fit a given problem.
This makes us very very efficient at anything that can be tackled with those simple programs, but also limit us in our ability to transcend them. If a superintelligence had to go beyond that, and explore higher regions in problem-space, would it be by developing a richer or more complex collection of pieces (and then, in turn, be limited by them)? Or would it do so by completely transcending the mechanism altogether and just having a really good way to build hypotheses from a true universal prior? One can assume for example there is a correct prior to formulate physics hypotheses, and ultimately everything is grounded in physics, but even for an ASI, having to think its way up from atoms every time it wants to understand humans seems a bit much. Yet having specialised human-understanding priors would in turn prejudice its ability to understand some hitherto unknown alien. Does intelligence even exist without this trade-off?
That's a puzzle you don't solve in four minutes with two attempts.