Another potential issue with search to me it seems less interpretable due to lacking easy methods of compression. I'm not sure how good the following analogy would be, but in chess, usually one cannot verify a N node search using significantly less than N nodes, since being able to do so would imply the original search was highly inefficient.
However, it could very well be the case that the real world is less like chess and verifying the consequences of executing some plan is significantly easier than finding the plan in the first place, in which case my concern isn't as strong.
I have actually been independently trying to interpret ARC's agenda for the case of NNUE (which I also expect to be quite difficult in general, as it is effectively a MLP with binary inputs), so this comment is useful clarification.
Some of these ambitious goals seem to be referring to the combined system of chess AI (search + evaluation), and I think search (especially minimax-based search) poses a much different flavor of interpretability compared to interpreting the raw NN based evaluation. Personally, I think the behavior of search alone is still extre... (read more)