If you or anyone else is interested in getting funding for this kind of threat modeling, Georgetown‘s Center for Security and Emerging Technology put out a RFP for risks of internally deployed frontier models due June 30.

https://cset.georgetown.edu/wp-content/uploads/FRG-Call-for-Research-Ideas-Internal-Deployment.pdf  

First, when I talk to security staff at AI companies about computer security, they often seem to fail to anticipate what insider threat from AIs will be like.

Why do you think this? Is it that they are not thinking about large numbers of automated agents running around doing a bunch of research?

Or is it that they are thinking about these kinds of scenarios, and yet they still don't apply the insider threat frame for some reason?

Have you taken a look at how companies manage Claude Code, Cursor, etc? That seems related.

The other consideration is do you also isolate the AI workers from the human insiders? Because you would still want to control scenarios where the AI has access to the humans that have access to sensitive systems. 

I agree insider vs outsider threat is an important distinction, and I one that I have seen security people take seriously in other contexts.  My background is in enterprise IT and systems admin. I think there's some practical nuance missing here.

In so far as security people are expecting to treat the AI as an outsider, they're likely expecting to have a hard boundary between "systems that run the AI" and "systems and tools the AI gets to use", where access to any given user is to only one or the other.  

This is already fairly common practice, in the sense that security people already think about how to avoid giving too much privileged access to any specific insider.  E.g. ideally in a large enough company, admins that have to most of the systems might not be given access to the systems responsible for logging what they did, nor the system granting access to other systems - centralized access control/identity (e.g. AWS IAM, Okta, etc.).  In that example the team that manages those logging and access control systems would not have access to anything outside those systems and would just be responsible for implementation according to whatever the security (and indirectly, HR people, say).  The security people might have just readonly access to systems (e.g. through a log search/archival tool and any telemetry from the security software) and rely on the separate IT team for all changes.

More extreme levels of security/paranoia are possible and do exist, especially if you look at the largest companies or national security applications.  In the latter case, the US govt maintains and entire separate internet with distinct fiber optic cabling, routers, etc; the computers have no connection outside this separate network, sit exclusively in shielded rooms in physically access controlled buildings, no unauthorized electronics go in those rooms, nor anyone without a clearance, etc.  I could go on but hopefully I've gotten the idea across; AI is a distinct threat that would need a different flavor of extreme precautions.

For AI there's a pretty natural boundary to draw between the AI specific hardware accelerators on which the AI training and inference runs (and associated control, data warehouse, etc), vs the tooling the AI is permitted to use while it's running.  The AI's inference running on a particular system doesn't automatically give it privileged access to that system (though exploits might).  Tooling that the AI calls into or uses need not exist on the same hardware that's running inference.  If such a boundary gets defined by the security team, it's then possible to treat any leakage of AI state effecting the inference hardware as fishy and deserving of investigation, just as if it was an outside user (b/c with that division, it is).

Happy to answer related questions about what existing approaches exist here and might be considered ordinary practice.  IT and information security mostly happens outside of academic literature and companies don't like to talk about specific security precautions they're taking.  You'll probably need to talk to many people with industry experience to get a well rounded take, as nobody gets to see all that many implementations in full detail in one career.

While you could give your internal AI wide indiscriminate access, it seems neither necessary nor wise to do so. It seems likely you could get at least 80% of the potential benefit via no more than 20% of the access breadth. I would want my AI to tell me when it thinks it could help me more with greater access so that I can decide whether the requested additional access is reasonable. 

I like the way you're thinking about AI and security. The insider vs outsider risk dynamic is very ingrained into Cybersecurity and is often discussed while threat modeling a system. The distinction is muddled because the most impactful external threat is one that steals an insiders credentials. History shows it's only a matter of time before insider permissions are used by an outsider, which gave birth to the phrase "assume breach". If the AI is a black-box, it's safest to assume it has been compromised. I'd like to add another dynamic, malicious vs accidental, to your AI insider risk model that makes clear why AIs should not be granted Insider privileges like human employees.

An insider has the ability to cause damage maliciously or accidentally. Accidental damage is by far the most common event, but we have decades, centuries if you count other engineering disciplines, of experience putting processes in place to prevent and learn from accidents. Generally the control mechanism slows the person down by asking for social proof, another person's sign-off, and that leads to a key mitigating control on their access: it's hard to cause major damage when action is slow. 

Adding social proof adds transparency, covers knowledge gaps and increases social pressure to get it right. People align with the goals of their group and intrinsically do not want to damage their group. When they harm the group accidentally it has a social cost, which acts as a preventative control. 

AIs act quickly and do not pay the same social cost. So those preventative controls do not count towards balancing the risk of overly broad insider permissions. 

Until we have Explainable AI, it will be risky to fully trust that an AI is free of hidden malicious intent or accident prone edge cases. The breadth of their input space makes firewalls inefficient and the speed they can act makes detective controls ineffective. 

AIs speed, lack of human-like social consequences and opaque decision making create a large source of risk that puts AI outside of what qualifies employees for insider privileges. Similarly their speed, clone-ability, and adaptability mean they don't need to work like humans and don't need insider privileges.

An AI can lookup and read documentation to acquire context faster than any person. It can be trained to do a new task (at least ones it's capable of) almost instantly. It can pass context to another AI much faster than a person. It can request permissions on a per task basis for specific rows/columns/fields in a database with minimal slowdown. An AI doesn't need to see the bigger picture, have a connection to the group or have novelty to stay motivated the way a person does. A person would quit if their job was to do the same step in a workflow 1000 times in a row and ask for permission with a unique identifier each time, so we built workflows to maintain employee happiness.

Separation of concerns, minimizing blast radius, observability and compose-ability are vital to building resilient software services. AIs do not change those principles. Using one AI for an entire workflow or the same AI across different workflows is not necessary the way it is for a person. So giving them permissions the way we do people is not necessary. Running instances of multiple different 600B+ parameter models may be prohibitively expensive today, but eventually it will be cost effective to have single use model instances in a similar style as AWS's Lambdas.

A natural extension of the way AI interacts today via MCP protocols makes it a kind of insider.  One with a specific role, and specific access patterns that match this role.

Even an org that is not concerned with misaligned AI or such, will still want to lock down exactly what updates each role is provding within the org, just as these org's typically lock down access to different roles within a company today.

Most employees cannot access account receivable, and access to the production databases in a tech company are very carefully guarded.   Mostly not from fear of malevolence, its just a fear that a junior Dev could easily bollix things horribly with one errant command. Much in the same way, the org will want to specialize the AI into different roles, and provide different access according to these roles, and will want to test these different AIs in each role.

All of this seems to follow quite naturally from existing corporate practice.

But I expect this level of diligence will fall short of something that could really stop a misaligned ASI or even strong AI.

So it seems this will be most like an insider threat, but I think the real remediation of this threat is far from clear.