npg

For what it's worth I haven't downvoted you. I generally agree with what you're saying. It's just that it seems basically impossible to come up with a coherent notion of "functionally identical" that goes beyond very conservative practical heuristics. It generally seems that disparate human beings function the same way, whereas I would certainly not say that LLMs function like humans at all. You might consider brain emulations instead, but I don't see any way to make up a rule that treats brain emulations and human brains on an equal footing as "computatio... (read more)

The hard problem of consciousness does not need to be solved in order for us to believe with high confidence that other people are conscious. Humans (and to a large extent animals) are extremely similar in origin and function, so it's perfectly reasonable to abduct that all of us are conscious. There is no such grounding to make such abductions about an agent that comes about in a completely different way like an AI model, because there isn't a clear relationship between consciousness and agency. I do not think people outside of certain communities will ever take the concept of AI consciousness seriously.

I expect, when offscript on openended problems, they won't be very good at finding new concept-handles that actually help them. But, when I imagine a 10,000-thread swarm solving Millenium problems, it does seem like "probe the concept-space and see if any of it feels helpful to crystallize into a new concept, and see if that concept helps" is the sorta thing they outta be able to do?

I don't think models actually have the ability to "probe the concept space" in the way that humans do, and that this is responsible for some of the limitations that you've obse... (read more)

I would say that I do see a gap in what attention mechanisms are capable of, and that this is indeed caused by limitations of the data flow structure. I also think that the examples of LLMs seeming to develop capabilities that people did not expect them to develop are largely explained by a vast increase in subject specific data incorporated into training, rather than actually developing general capabilities. The failure modes of LLMs have not really changed in character. It's true that they occur less frequently, especially in the domains that have been targeted most heavily by the corporations. But they haven't gone away, and are still prevalent in many disciplines.

As a mathematician I don't think there's some kind of qualitatively different "creativity" other than being able to notice and track high-order patterns across different kinds of data and different levels of abstraction.

As a theoretical physicist I also think this is basically true but that the models seems to remain much worse at this than humans. With a particular emphasis on "high-order".

However they do have much more data to work from and, as you say, massive persistence. As far as I can tell the discoveries that have come out of them so far have come... (read more)

I don't think you're splitting complexity correctly. ICLing a high level abstract is not inherently more complex than ICLing a highly complex syntactic rule.

I was talking about difficulty, not complexity. These are not the same thing. Syntactic rules can in principle be given arbitrary complexity, but one has a clear set of instructions to follow in order to implement them. As long as an LLM can implement the steps of a rule, it should in principle be able to compose them sequentially by tracking the sequence with attention. Whether or not it can do so in... (read more)

In domains where "human priors" don't apply, sample efficiency of human learning takes a dive. See "Investigating Human Priors for Playing Video Games" for an instructive example - removing just the "surface level visual cues" from the games without changing the game logic dropped human performance by an order of magnitude. Goes to show that a lot of "human advantage" was resting on well made assumptions.

This paper is really not the takedown of human sample efficiency that you're making it out to be. The conclusion is that when you deliberately confuse the... (read more)

It seems that we disagree about how powerful ICL actually is.

I agree with what you're saying about ICL+distillation not being that different from RL in spirit. I just also believe that both of these are pretty limited in terms of what they can actually teach an LLM to do without appropriate priors encoded in the base model. This doesn't apply to humans! It seems really important that as we generate a trajectory this backreacts on our priors in a way that reshapes the future of that trajectory. This means that the trajectories we generate are way less limi... (read more)

Sorry, didn't see this before. Yeah, pretty much. Although I actually believe that any frozen model architecture is limited by this kind of creativity wall. On the other hand, I suspect that a humanlike learning algorithm would allow for more genuine creative exploration even without external data inputs, because it would be able to interpret the weights and find interesting conceptual directions within them. I don't see any reason to think that such technology is on the horizon, but it is much much scarier (even in the absence of RSI in the classical sense).

I agree with the first part of your comment, but about "in-context learning" - this is just really shallow to the point that I don't believe it's worth comparing to a bona fide "learning algorithm" at all. It's just what transformers do; every time they generate a new token, influences are propagated from prior generated tokens according to pathways learned by the model during training. If you provide the model a list of instructions, the attention mechanism will impart that information into later latent vectors, which condition the output tokens in such a... (read more)

AlphaZero gets a bunch of data on what works and what doesn't work from Monte Carlo Tree Search. It's running a bunch of experiments on chess or Go and slowly absorbing the results into the weights of the neural net. In the case of RL on informal tasks, there is no such external automated source of data. The model has to produce a bunch of attempts on any given task, and if none of those attempts succeed, it doesn't get any learning signal. The only "exploration" comes from tiny perturbations to the model weights, so if the model just doesn't have the idea... (read more)

I can think of a simple test that I think is beyond the abilities of the LLM paradigm and isn't close to the extremes of "inventing fields of science" or "massively successful entrepreneurship". Unfortunately it can't be applied directly to commercial LLMs. Hold out any and all data on some conceptually deep subject --- for example, general relativity --- from training, but otherwise allow the model to be as big and well-trained as required. Then give the model access to one textbook on the subject. Let it do whatever it wants with that textbook. Then assi... (read more)

There appears to be some intuitive distinction between math results that are just "recombinations of known ideas" (possibly very sophisticated recombinations!), and math results that feel "truly novel", and the mathematical consensus is apparently that none of AI math results so far are of the latter type.

I think the conclusion is correct here but the model is not quite right. Most (or all) ideas originate from previous concepts and information, passed through a variable amount of reprocessing and abstraction. People understood the concepts of slope and ar... (read more)