cfoster0

Agreed! I may try to adopt your terminology. Have also seen folks use "cheating" instead of "task gaming".

Here's a footnote from the METR post discussing this distinction:

It’s common to use the term “reward hacking” to refer to any way models “cheat” on tasks, and this blogpost will use that convention. However, we think it’s helpful for conceptual clarity to remember the model doesn’t receive any reward corresponding to its scores on our evaluations. Rather, its behavior on our evaluations is evidence that it learned to “cheat” in training to receive hi

Please work on this! Very interested in results from this technique and others like it.

A several thousand dimensional vector can contain so much more information than is in an integer between 1 and ~200K. The implementation is likely painful, but I can't see a world where the optimal bandwidth given a good implementation of both is lower

The transformer already has thousands of dimensions available through attention, no? How much does removing the tokenization buy you in addition? I agree it buys you some but seems unclear how much.

Note that many of these same weird tokens have been observed in GPT-5 chains-of-thought (at least "marinade", "illusions", "overshadow").

Yes, I think that what it takes to advance the AI capability frontier has changed significantly over time, and I expect this to continue. That said, I don’t think that existing algorithmic progress is irrelevant to powerful AI. The gains accumulate, even though we need increasing resources to keep them coming.

AFAICT, it is not unusual for productivity models to account for stuff like this. Jones (1995) includes it in his semi-endogenous growth model where, as useful innovations are accumulated, the rate at which each unit of R&D effort accumulates more... (read more)

Researchers have had (and even published!) tons of ideas that looked promising for smaller tasks and smaller budgets but then failed to provide gains—or hurt more than they help—at larger scales, when combined with their existing stuff. That’s why frontier AI developers “prove out” new stuff in settings that are close to the one they actually care about. [1]

Here’s an excerpt from Dwarkesh’s interview with Sholto and Trenton, where they allude to this:

Sholto Douglas 00:40:32

So concretely, what does a day look like? I think the most important part to illustr

Like I think the view would have to be that "frontier scale" varied along with the 7 OOMs of compute difference, but I'm not sure I buy this.

Wait, why not? I’d expect that the compute required for frontier-relevant experimentation has scaled with larger frontier training runs.

Other proponents of the bill (longform, 1-3h)

[...]

Charles Foster

Note: I wouldn't personally call myself a proponent, but I'm fine with Michaël putting me in that bucket for the sake of this post.

I’m not sure if you intended the allusion to “the tendentious assumption in the other comment thread that courts are maximally adversarial processes bent on on misreading legislation to achieve their perverted ends”, but if it was aimed at the thread I commented on… what? IMO it is fair game to call out as false the claim that

It only counts if the $500m comes from "cyber attacks on critical infrastructure" or "with limited human oversight, intervention, or supervision....results in death, great bodily injury, property damage, or property loss."

even if deep... (read more)

If you read the definition of critical harms, you’ll see the $500m doesn’t have to come in one of those two forms. It can also be “Other grave harms to public safety and security that are of comparable severity”.

I was trying to write a comment to explain my reaction above, but this comment said everything I would have said, in better words.

OK, in case this wasn't clear: if you are a Californian and think this bill should become law, don't let my comment excuse you from heeding the above call to action. Contacting your representatives will potentially help move the needle.

Unfortunately, due to misinformation and lobbying by big tech companies, SB 1047 is currently stalled in the Assembly Appropriations Committee.

This is extremely misleading. Any bill that would have non-negligible fiscal impact (the threshold is only $150,000 https://apro.assembly.ca.gov/welcome-committee-appropriations/appropriations-committee-rules) must be put in the Appropriation Committee “Suspense File” until after the budget is prepared. That is the status of SB 1047 and many many other bills. It has nothing to do with misinformation or lobbying, ... (read more)

What's the evidence that this document is real / written by Anthropic?

Axios first reported on the letter, quoting from it but not sharing it directly:

https://www.axios.com/2024/07/25/exclusive-anthropic-weighs-in-on-california-ai-bill

The public link is from the San Francisco Chronicle, which is also visible in the metadata on the page citing the letter as “Contributed by San Francisco Chronicle (Hearst Newspapers)”.

https://www.sfchronicle.com/tech/article/wiener-defends-ai-bill-tech-industry-criticism-19596494.php

Left the following comment on the blog:

I appreciate that you’re endorsing these changes in response to the two specific cases I raised on X (unlimited model retraining and composition with unsafe covered models). My gut sense is still that ad-hoc patching in this manner just isn’t a robust way to deal with the underlying issue*, and that there are likely still more cases like those two. In my opinion it would be better for the bill to adopt a different framework with respect to hazardous capabilities from post-training modifications (something closer to “C

You want to learn an embedding of the opportunities you have in a given state (or for a given state-action), rather than just its potential rewards. Rewards are too sparse of a signal.

More formally, let's say instead of the Q function, we consider what I would call the Hope function: which given a state-action pair (s, a), gives you a distribution over states it expects to visit, weighted by the rewards it will get. This can still be phrased using the Bellman equation:

Hope(s, a) = rs' + f Hope(s', a')

The "successor representation" is somewhat close to this... (read more)

On reflection these were bad thresholds, should have used maybe 20 years and a risk level of 5%, and likely better defined transformational. The correlation is certainly clear here, the upper right quadrant is clearly the least popular, but I do not think the 4% here is lizardman constant.

Wait, what? Correlation between what and what? 20% of your respondents chose the upper right quadrant (transformational/safe). You meant the lower left quadrant, right?

Very surprised there's no mention here of Hanson's "Foom Liability" proposal: https://www.overcomingbias.com/p/foom-liability

I appreciate that you are putting thought into this. Overall I think that "making the world more robust to the technologies we have" is a good direction.

In practice, how does this play out?

Depending on the exact requirements, I think this would most likely amount to an effective ban on future open-sourcing of generalist AI models like Llama2 even when they are far behind the frontier. Three reasons that come to mind:

1. The set of possible avenues for "novel harms" is enormous, especially if the evaluation involves "the ability to finetune [...], external tool

If we are assessing the impact of open-sourcing LLMs, it seems like the most relevant counterfactual is the "no open-source LLM" one, right?