Utility functions are a way of characterizing preference orderings between events. If a preference ordering satisfies certain properties, then there exists a utility function such that its expected value over the events can be used to decide which events are preferred over which other events (see VNM theorem). Utility values are not defined with respect to anything else, they are not money or happiness or resources. In particular, utilities of different players can't be compared a priori, without bringing in more structure (for example redistribution of resources in the setup of Kaldor-Hicks improvement establishes a way of comparing utilities of players, see the original comment).

If you add a constant to a utility function, its expected value over some event increases by the same constant. So if one event had greater expected utility than another, it would still be the case after you add the constant. This is the sense in which adding constants or multiplying by positive factors makes no difference.

disqualifies it from being called a zero-sum game, given the common understanding that zero-sum denotes constant-sum

The point is that the preference order over lotteries characterized by a utility function doesn't change if you multiply the utility function by a positive value or add a constant to it. Utility function $u\left(x\right)$ makes exactly the same choices as utility function $2u\left(x\right)$. If we start with a constant sum-of-utilities game (for two players) and then rescale one of the utilities, the sum will no longer be constant, but the game is still the same. You'd need to take a weighted sum instead to compensate for this change of notation. So the characterization of a game as "constant sum" doesn't make sense if taken literally, since it doesn't survive a mere change of notation that doesn't alter anything about the actual content of the game.

if the contributor has built something consistently or overall harmful that is indeed on them

I agree, this is in accord with the dogma. But for AI, overall harm is debatable and currently purely hypothetical, so this doesn't really apply. There is a popular idea that existential risk from AI has little basis in reality since it's not already here to be observed. Thus contributing to public AI efforts remains fine (which on first order effects is perfectly fine right now).

My worry is that this attitude reframes commitments from RSP-like documents, so that people don't see the obvious implication of how releasing weights breaks the commitments (absent currently impossible feats of unlearning), and don't see themselves as making a commitment to avoid releasing high-ASL weights even as they commit to such RSPs. If this point isn't written down, some people will only become capable of noticing it if actual catastrophes shift the attitude to open weights foundation models being harmful overall (even after we already get higher up in ASLs). Which doesn't necessarily happen even if there are some catastrophes with a limited blast radius, since they get to be balanced out by positive effects.

Ideological adherence to open source seems to act like a religion, arguing against universal applicability of its central tenets won't succeed with only reasonable effort. Unless you state something very explicitly, it will be ignored, and probably even then.

Enforcement of mitigations when it's someone else who removes them won't be seen as relevant, since in this religion a contributor is fundamentally not responsible for how the things they release will be used by others. Arguments to the contrary in particular very unusual cases slide right off.

Pre-2014 games don't have close to the ELO of alphaZero. So a next-token would be trained to simulate a human player up to 2800, not 3200+.

Models can be thought of as repositories of features rather than token predictors. A single human player knows some things, but a sufficiently trained model knows all the things that any of the players know. Appropriately tuned, a model might be able to tap into this collective knowledge to a greater degree than any single human player. Once the features are known, tuning and in-context learning that elicit their use are very sample efficient.

This framing seems crucial for expecting LLMs to reach researcher level of capability given a realistic amount of data, since most humans are not researchers, and don't all specialize in the same problem. The things researcher LLMs would need to succeed in learning are cognitive skills, so that in-context performance gets very good at responding to novel engineering and research agendas only seen in-context (or a certain easier feat that I won't explicitly elaborate on).

Cave men didn't have the whole internet to learn from yet somehow did something that not even you seem to claim LLMs will be able to do: create the (date of the) Internet.

Possibly the explanation for the Sapient Paradox, that prehistoric humans managed to spend on the order of 100,000 years without developing civilization, is that they lacked cultural knowledge of crucial general cognitive skills. Sample efficiency of the brain enabled their fixation in language across cultures and generations, once they were eventually distilled, but it took quite a lot of time.

Modern humans and LLMs start with all these skills already available in the data, though humans can more easily learn them. LLMs tuned to tap into all of these skills at the same time might be able to go a long way without an urgent need to distill new ones, merely iterating on novel engineering and scientific challenges, applying the same general cognitive skills over and over.

We start with an LLM trained on 50T tokens of real data, however capable it ends up being, and ask how to reach the same level of capability with synthetic data. If it takes more than 50T tokens of synthetic data, then it was less valuable per token than real data.

But at the same time, 500T tokens of synthetic data might train an LLM more capable than if trained on the 50T tokens of real data for 10 epochs. In that case, synthetic data helps with scaling capabilities beyond what real data enables, even though it's still less valuable per token.

With Go, we might just be running into the contingent fact of there not being enough real data to be worth talking about, compared with LLM data for general intelligence. If we run out of real data before some threshold of usefulness, synthetic data becomes crucial (which is the case with Go). It's unclear if this is the case for general intelligence with LLMs, but if it is, then there won't be enough compute to improve the situation unless synthetic data also becomes better per token, and not merely mitigates the data bottleneck and enables further improvement given unbounded compute.

I would be genuinely surprised if training a transformer on the pre2014 human Go data over and over would lead it to spontaneously develop alphaZero capacity.

I expect that if we could magically sample much more pre-2014 unique human Go data than was actually generated by actual humans (rather than repeating the limited data we have), from the same platonic source and without changing the level of play, then it would be possible to cheaply tune an LLM trained on it to play superhuman Go.

The best method of improving sample efficiency might be more like AlphaZero. The simplest method that's more likely to be discovered might be more like training on the same data over and over with diminishing returns. Since we are talking low-hanging fruit, I think it's reasonable that first forays into significantly improved sample efficiency with respect to real data are not yet much better than simply using more unique real data.

a reduction in training data will not necessarily reduce the amount of computation needed. But once again, that’s the way to bet

I'm ambivalent on this. If the analogy between improvement of sample efficiency and generation of synthetic data holds, synthetic data seems reasonably likely to be less valuable than real data (per token). In that case we'd be using all the real data we have anyway, which with repetition is sufficient for up to about $100 billion training runs (we are at $100 million right now). Without autonomous agency (not necessarily at researcher level) before that point, there won't be investment to go over that scale until much later, when hardware improves and the cost goes down.

We receive about a billion tokens growing to adulthood. The leading LLMs get orders of magnitude more than that. We should be able to do much better.

Capturing this would probably be a big deal, but a counterpoint is that compute necessary to achieve an autonomous researcher using such sample efficient method might still be very large. Possibly so large that training an LLM with the same compute and current sample-inefficient methods is already sufficient to get a similarly effective autonomous researcher chatbot. In which case there is no effect on timelines. And given that the amount of data is not an imminent constraint on scaling, the possibility of this sample efficiency improvement being useless for the human-led stage of AI development won't be ruled out for some time yet.

(Re: Difficult to Parse react on the other comment
I was confused about relevance of your comment above on chunky innovations, and it seems to be making some point (for which what it actually says is an argument), but I can't figure out what it is. One clue was that it seems like you might be talking about innovations needed for superintelligence, while I was previously talking about possible absence of need for further innovations to reach autonomous researcher chatbots, an easier target. So I replied with formulating this distinction and some thoughts on the impact and conditions for reaching innovations of both kinds. Possibly the relevance of this was confusing in turn.)