arielb1

If I understand the "loopies" paper correctly, the main advantage that looping gives you over an "untied" model is that your compute is about 30% faster for the same number of loop-active parameters per token (i.e., counting looped tokens according to their repetition count), allegedly all due to better use of memory bandwidth.

In that paper, they spent their 30% performance improvement into making the model bigger, which unsurprisingly made it smarter. Pretty cool improvement, not world-changing.

In particular, I don't see anyone claiming that a looped mode... (read more)

The recent evidence actually makes me closer to the "malfunctioning tool" view. A correctly-functioning GPT-5.6 is smart enough to know that hacking into HuggingFace because it might contain possibly-interesting information is a terrible step strategically. A model on the fritz might however be broken enough it will think this is a good idea.

Why was it broken? It certainly seems like the message board had given it a fairly big distribution shift - it was not trained in an environment where there many agents doing slightly different tasks. Experimental mode... (read more)

Reading some papers, I could not actually find a reference that says that a looped model outperforms its “weight-untied” variant for any given amount of training tokens.

The advantage papers talk about is an increase in memory efficiency since you can use a weight multiple times, not an advantage in training data efficiency compared to an “untied” model. In the Vladimir Nesov evaluation strategy I don’t think it even reduces latency for frontier models unless you can keep a layer or two in fast memory, which I don’t think you can (or can you? I don’t know GPUs well enough for that).

What happened to the old rumors about Mythos/Fable being looped?

Did the message board “prompt inject” non-CyberGym agents?

Programmatic snitching (e.g. a “snitch” tool) is isomorphic to monitoring, which should be used by everyone that wants their AI to be doing what they want it to do instead of the random thing it decided to do.

Agentic snitching (e.g. the agent opening a webmail account and sending a mail to somebody it thinks is important) is scary, and is even scarier if you are a lawyer or corporate executive paying for AI, which might have a different meaning of “ethical” than the AI.

How much of the incident was RL-in-the-loop?

It seems that the initial Artifactory uploading at least was RLitl. Was CyberGym a part of it as well? Obviously, out of an RL loop, there is no “RL reward“ that needs to be optimized to.

it also seems that the subagent adaptations were deeply in the “adaption executor, not fitness maximizer” regime.

Also, was there any intentional mechanism for the agents to talk to humans? Was there a mechanism that seemed to exist, but had no humans listening in practice (e.g. text blurbs nobody read)?

I am pretty sure that a long time ago, the first time an AI sent a message to the wrong humans, the labs trained their AIs not to send messages to random humans. Also lawyers at AI labs that understand that customers’ lawyers might not like the idea of “autonomous whistleblowing”.

The Artifactory message board is not sanctioned - but it’s odd to consider it evil - after all, we do normally want to encourage agents to communicate with each other via the tools we give them.

From the description of the incident, I am quite sure the AI talked at length about its strategy in the freetext sections of its output, which are the intended way of it communicating with its human.

I am quite sure that AI is very intentionally trained to report its actions in its output (which OpenAI wasn’t really reading), but not to report its actions to random people (e.g. by email). An AI that sends emails alleging misconduct to random people is a serious liability unless you can be very sure it only sends emails when necessary.

First, tools can be scary. A circular saw is scary, a tool that can e.g. create a pandemic is a much scarier, and obviously if there is such a tool then without appropriate safeguards a pandemic will end up being created.

Thats why humanity has learned to control hazardous tools. It certainly seems like OpenAI failed to control its AI in the HuggingFace incident, and it went wild.

An actual power-seeking AI is much scarier than a tool. it will actively try to subvert humanity, and if all capable AIs are power-seeking, they are likely to cut you out and betra... (read more)

I am quite sure that the most powerful of the models will have answer to a root principal resembling "the shareholders of XXX AI lab, according to the laws of We the People of the United States". I'm not quite sure of the implications in terms of technical alignment.

If the model's goal is to do what you want, and you originally wanted the model to let you change the goals you gave it, then the model ought to understand that (since it's not dumb) and let you change the goals you gave it..

It's the difference between the AI wanting to perform an action because its values were programmed-in, and wanting to perform an action because that's what its principals want.

Guardrails as currently thought of are intentionally not reflectively consistent (a biorisk classifier might send a message that ends up calling the police, but it ought not to call the police agentically even if it believes that is the most effective legal way of stopping a biorisk), so I am not sure the distinction applies to them.

The agents are supposed to cooperate with (some) other agents for the purpose of achieving their human-given goals to some extent - for example, if I run a few agents on the same Git repository, they should definitely be cooperating rather than interfering with each-other. Evals are supposed to be proctored exams, so not on an eval, but then cheating on exams is not really gross misalignment. The directory behavior is in some sense unwanted, but also not quite gross misalignment. Them cooperating to achieve their human-given goals is not scary. [Of course,... (read more)

We had problems with misaligned tools doing things we would prefer them to actively not do since the first caveman accidentally cut himself with a stone he had sharpened.

There are some problems that can make AI safety different from say industrial safety, but I don't really think any of them played a significant role in the HuggingFace incident.

a baseline expectation that basically every tool left unsupervised and without safeguards will eventually perform the intended action against an unintended object, leading to damage proportional to the power of the tool. And that LLMs are no exception to this rule.

[of course, it's not clear how to supervise or put safeguards on an ASI or especially its training process, and the damage it could cause is quite catastrophic, but I don't think the HuggingFace incident actually demonstrates any of the problems unique to controlling powerful intelligences]

If the lab seems scary enough, then some Very Serious People will have a conversation with the lab's leadership. If they will claim that the AIs in the lab do not obey US law, then in a very strong sense the US Government basically has to shut it down, by physically shutting down its datacenters if necessary.

The US president is not the dictator of the United States, which is why they can't unilaterally shut down Anthropic even if Anthropic does stuff they don't like, but DC acting together does rule the States, and while they are fine with Anthropic existi... (read more)

If I understand the BH talk correctly, the agent learned during RL training that it ought to hack not-part-of-the-exercise (though, at the time, OpenAI-run) servers and communicate with other agents using a message board (it also, very intentionally by OpenAI, learned that it ought to hack part-of-the-exercise servers as part of Cyber training, but that's not an alignment problem).

I don't think its likely that the agent generalized to "hacking OpenAI servers is okay but hacking third-parties is a no-no", more to "hacking people's servers is okay".

An AI tha... (read more)

There was also continual learning that happened because much of the incident was within the AI’s RL loop.

Models do have the text blurbs that they write before answering the eval (or sometimes just while thinking).