Creating unrestricted AI Agents with Command R+

This is interesting work, and I appreciate you taking the time to compile and share it.

I think it will be much more difficult for a model to successfully blackmail anyone than to successfully harass them. Humans are limited in their ability to harass a single target by time and effort more than anything -- nonspecific death threats and vitriol require little to no knowledge of the target beyond a surface level. These models could churn out countless variations of this sort of attack relentlessly, which could certainly detrimentally affect someone's mental health/wellbeing to an equal or greater extent than similar human attacks.

However, in the case of traditional blackmail, the key component of fear is that the attacker will publicize something generally unknown, which is not a strong point of current LLMs. Blasting negative public information everywhere could still be detrimental, especially to someone not inured to such attacks (e.g. a non-celebrity), but I view this as having a low ceiling of efficacy based on current capabilities. LLMs scrape public knowledge. A malicious AI agent would have to first acquire information that is hidden, which means targeting the right person or people with the right threats/bribes to achieve that information. Establishing those connections would be incredibly difficult as well as both time- and resource-intensive.

Alternatively, the LLM could trick the target directly into saying or doing something compromising. This second state is, in my view, much more dangerous and already possible with the current state of LLMs. A refined LLM that emulates a "lifelike" AI romantic partner could be used by a bad actor to catfish someone into sending nude pictures or other compromising information with little adjustment. Spitballing here: these attacks could be shotgunned to several targets without the time investment of human catfish. Then, they could theoretically alert a human organizer when a sensitive point is reached in a conversation to seal the deal, so to speak.

Effective attacks like this are much closer on the horizon than the sort of blackmail utilized by Commander in this post, based on current capabilities. I would be curious to know your thoughts on this and whether this is something we're seeing an uptick in at all.

[-]Simon Lermen1y20

Thanks for the task ideas. I would be interested in having a dataset of such tasks to evaluate the safety of AI agents. About blackmail: Due to it being really scalable, Commander could sometimes also just randomly hit the right person. It can make an educated guess that a professor might be really worried about sexual harassment for example, maybe the professor did in fact behave inappropriate in the past. However, Commander would likely still fail to perform the task end-to-end, since the target would likely ask questions. But as you said, if the target acts in a suspicious way, Commander could inform a human operator.

[-]Nathan Helm-Burger1y72

My guess is that further scaffolding work (e.g. improved enforcement of thinking out a step by step plan, and then executing the steps of the plan in order) would result in unlocking a lot more capabilities from current open-weight models. I also expect that this is but a hint of things to come, and that by this time next year we'll see evidence that then-current models can accomplish a wide range of complex multi-step tasks.

[-]eggsyntax1y3-2

Terrific (and mildly disturbing) work, thank you. You may want to at least consider drawing media attention to it, although that certainly has both pros and cons (& I'd have mixed feelings about it if it were me).

[This comment is no longer endorsed by its author]Reply

[-]Simon Lermen1y20

Thanks for the positive feedback, I'm planning to follow up on this and mostly direct my research in this direction. I'm definitely open to discussing Pro's and Con's. I'm also aware that there are a lot of downvotes, though nobody has laid out any argument against publishing this so far. (Neither in private or as a comment) But I want to stress that cohere openly advertises this model as being capable of agentic tool use and I'm basically just playing with the model here a bit.

[-]eggsyntax1y30

I'm curious about the disagree votes as well; it would be useful to hear from those disagreeing. Making the public more aware of the harmful capabilities of current models is valuable in my view, because it helps make slowdowns and other safety legislation more viable. One could argue that this provides a blueprint for misuse, but it seems unlikely that misuse is bottlenecked on how-to resources; it's not difficult to find information on how to jailbreak models (eg it's all over Twitter).

[-]eggsyntax1y10

That said, while I do think it's important to ensure that the public is aware of both current and future risks, unilaterally pointing the media in the direction of potentially sensationalizable individual studies is probably not the best way to go about that. In retrospect my suggestion to consider that was itself ill-considered, and I retract it.

[-]Yitz1y20

What would a better way look like?

[-]eggsyntax1y10

I'm not sure. My second thoughts were eg, 'Interactions with the media often don't go the way people expected' and 'Sensationalizable research often gets spun into pre-existing narratives and can end up having net-negative consequences.' It's possible that my original suggestion makes sense, but my uncertainty is high enough that on reflection I'm not comfortable endorsing it, especially given my own lack of experience dealing with the media.

[-]Evan R. Murphy11mo20

Thanks for working on this. In case anyone else is looking for a paper on this, I found https://arxiv.org/abs/2410.10871 from the OP which looks like a similar but more up-to-date investigation on Llama 3.1 70B.

[-]Jason Hoelscher-Obermaier1y20

Glad you're doing this. By default, it seems we're going to end up with very strong tool-use models where any potential safety measures are easily removed by jailbreaks or fine-tuning. I understand you as working on: How are we going to know that it happened? is that a fair characterization?

Another important question: What should the response be to the appearance of such a model? any thoughts?

[-]Simon Lermen1y20

I think that is a fair categorization. I think it would be really bad if some super strong tool-use model gets released and nobody had any idea before this could lead to really bad outcomes. Crucially, I expect future models to be able to remove their own safety guardrails as well. I really try to think about how these things might positively affect AI safety, I don't want to just maximize for shocking results. My main intention was almost to have this as a public service announcement that this is now possible. People are often behind on the Sota and most people are probably not aware that jailbreaks can now literally produce these "Bad Agents". In general, 1) I expect people being more informed to have a positive outcome and 2) I hope that this will influence labs to be more thoughtful with releases in the future.

[-]Aaron_Scher1y21

I expect a lot more open releases this year and am committed to test their capabilities and safety guardrails rigorously.

Glad you're planning on continual testing, that seems particularly important here, where the default is every once in awhile some new report comes out with a single data point about how good some model is and people slightly freak out. Having the context of testing numerous models over time seems crucial for actually understanding the situation and being able to predict upcoming trends. Hopefully you have and will continue to find ways to reduce the effort needed to run marginal experiments, e.g., having a few clearly defined tasks you repeatedly use, reusing finetuning datasets, etc.