Wiki Contributions



No one really knew why tokamaks were able to achieve such impressive results. The Soviets didn’t progress by building out detailed theory, but by simply following what seemed to work without understanding why. Rather than a detailed model of the underlying behavior of the plasma, progress on fusion began to take place by the application of “scaling laws,” empirical relationships between the size and shape of a tokamak and various measures of performance. Larger tokamaks performed better: the larger the tokamak, the larger the cloud of plasma, and the longer it would take a particle within that cloud to diffuse outside of containment. Double the radius of the tokamak, and confinement time might increase by a factor of four. With so many tokamaks of different configurations under construction, the contours of these scaling laws could be explored in depth: how they varied with shape, or magnetic field strength, or any other number of variables.


Hadn't come across this analogy to current LLMs. Source: This interesting article.


Case in point: This is a five year old tsne plot of word vectors on my laptop. 


I don't get what role the "gaps" are playing in this. 

Where is it important for what a tool is, that it is for a gap and not just any subproblem? Isn't a subproblem for which we have a tool never a gap? 

Or maybe the other way around: Aren't subproblem classes that we are not willing to leave as gaps those we create tools for? 

If I didn't know about screwdrivers I probably wouldn't say "well, I'll just figure out how to remove this very securely fastened metal thing from the other metal thing when I come to it".


I'd be very interested to learn more about how your research agenda has progressed since that first post.


The post about learned lookahead in Leela has kind of galvanised me into finally finishing an investigation I have worked on for too long already. (Partly because I think that finding is incorrect, but also because using Leela is a great idea, I had got stuck with LLMs requiring a full game for each puzzle position). 

I will ping you when I write it up. 


Mira Murati said publicly that "next gen models" will come out in 18 months, so your confidential source seems likely to be correct.


This is a fairly straightforward point, but one I haven't seen written up before and I've personally been wondering a bunch about.


I feel like I have been going on about this for years. Like here, here or here. But I'd be the first to admit, that I don't really do effort posts. 


Hmm, yeah, I think we are talking past each other. 

Everything you describe is just pattern recognition to me. Lookahead or search does not depend on the broadness of the motive. 

Lookahead, to me, is the ability to look ahead and see what is there. It allows very high certainty even for never before seen mating combinations. 

If the line is forcing enough it allows finding very deep combinations (which you will never ever find with pattern recognition because the combinatorial explosions means that basically every deep combination has never been seen before).

In humans, it is clearly different from pattern recognition. Humans can see multi-move patterns in a glance. The example in the post I would play instantly in every blitz game. I would check the conditions of the pattern, but I wouldn't have to "look ahead". 

Humans consider future moves even when intuitively assessing positions. "This should be winning, because I still have x,y and z in the position". But actually calculating is clearly different because it is effortful. You have to force yourself to do it (or at least I usually have to). You manipulate the position sequentially in your mind and see what could happen. This allows you to see many things that you couldn't predict from your past experience in similar positions

I didn't want to get hung up on whether there is a crisp boundary. Maybe you are right and you just keep generalising and generalising until there is a search algo in the limit. I very much doubt this is where the ability of humans to calculate ahead comes from. In transformers? Who knows. 


I don't think my argument relies on the existence of a crisp boundary. Just on the existence of a part of the spectrum that clearly is just pattern recognition and not lookahead but still leads to the observations you made. 

Here is one (thought) experiment to tease this apart: Imagine you train the model to predict whether a position leads to a forced checkmate and also the best move to make. You pick one tactical motive and erase it from the checkmate prediction part of the training set, but not the move prediction part. 

Now the model still knows which the right moves are to make i.e. it would play the checkmate variation in a game. But would it still be able to predict the checkmate? 

If it relies on pattern recognition it wouldn't - it has never seen this pattern be connected to mate-in-x. But if it relies on lookahead, where it leverages the ability to predict the correct moves and then assesses the final position then it would still be able to predict the mate. 

The results of this experiment would also be on a spectrum from 0% to 100% of correct checkmate-prediction for this tactical motive. But I think it would be fair to say that it hasn't really learned lookahead for 0% or a very low percentage and that's what I would expect.


I think the spectrum you describe is between pattern recognition by literal memorisation and pattern recognition building on general circuits. 

There are certainly general circuits that compute whether a certain square can be reached by a certain piece on a certain other square. 

But if in the entire training data there was never a case of a piece blocking the checkmate by rook h4, the  existence of a circuit that computes the information that the bishop on d2 can drop back to h6 is not going to help the "pattern recognition"-network to predict that Ng6 is not a feasible option. 

The "lookahead"-network however would go through these moves and assess that 2.Rh4 is not mate because of 2...Bh6. The lookahead algorithm would allow it to use general low-level circuits like "block mate", "move bishop/queen on a diagonal" to generalise to unseen combinations of patterns.  


I thought I spell this out a bit: 

What is the difference between multi-move pattern recognition and lookahead/search? 

Lookahead/search is a general algorithm, build on top of relatively easy-to-learn move prediction and position evaluation. In the case of lookahead this general algorithm takes the move prediction and goes through the positions that arise when making the predicted moves while assessing these positions. 

Multi-move pattern recognition starts out as simple pattern recognition: The network learns that Ng6 is often a likely move when the king is on h8, the queen or bishop takes away the g8 square and there is a rook or queen ready to move to the h-file. 

Sometimes it will predict this move although the combination, the multi-move tactical strike, doesn't quite work, but over time it will learn that Ng6 is unlikely when there is a black bishop on d2 ready to drop back to h6 or when there is a black knight on g2, guarding the square h4 where the rook would have to checkmate. 

This is what I call multi-move pattern recognition: The network has to learn a lot of details and conditions to predict when the tactical strike (a multi-move pattern) works and when it doesn't. In this case you would make the same observations that where described in this post: For example if you'd ablate the square h4, you'd lose the information of whether this will be available for the rook in the second move. It is important for the pattern recognition to know where future pieces have to go. 

But the crucial difference to lookahead or search is that this is not a general mechanism. Quite the contrary, it is the result of increasing specialisation on this particular type of position. If you'd remove a certain tactical pattern from the training data the NN would be unable to find it. 

It is exactly the ability of system 2 thinking to generalise much further than this that makes the question of whether transformers develop it so important. 

I think the methods described in this post are even in principle unable to distinguish between multi-move pattern recognition and lookahead/search. Certain information is necessary to make the correct prediction in certain kinds of positions. The fact that the network generally makes the correct prediction in these types of positions already tells you that this information must be processed and made available by the network. The difference between lookahead and multi-move pattern recognition is not whether this information is there but how it got there. 

Load More