npg

AlphaZero gets a bunch of data on what works and what doesn't work from Monte Carlo Tree Search. It's running a bunch of experiments on chess or Go and slowly absorbing the results into the weights of the neural net. In the case of RL on informal tasks, there is no such external automated source of data. The model has to produce a bunch of attempts on any given task, and if none of those attempts succeed, it doesn't get any learning signal. The only "exploration" comes from tiny perturbations to the model weights, so if the model just doesn't have the idea... (read more)

I can think of a simple test that I think is beyond the abilities of the LLM paradigm and isn't close to the extremes of "inventing fields of science" or "massively successful entrepreneurship". Unfortunately it can't be applied directly to commercial LLMs. Hold out any and all data on some conceptually deep subject --- for example, general relativity --- from training, but otherwise allow the model to be as big and well-trained as required. Then give the model access to one textbook on the subject. Let it do whatever it wants with that textbook. Then assi... (read more)

There appears to be some intuitive distinction between math results that are just "recombinations of known ideas" (possibly very sophisticated recombinations!), and math results that feel "truly novel", and the mathematical consensus is apparently that none of AI math results so far are of the latter type.

I think the conclusion is correct here but the model is not quite right. Most (or all) ideas originate from previous concepts and information, passed through a variable amount of reprocessing and abstraction. People understood the concepts of slope and ar... (read more)