Why doesn't OpenAI just test their models for now in airgapped environments (a physical building/location/server entirely disconnected from the internet)?
Reports about new discoveries of agents finding clever ways to escape containment are still rolling in by the day, post-HF incident. The latest (to my knowledge) occurred just this week: https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot/
So until we actually know how to build better, more robust sandboxes, and prove it, doesn't it just make sense to accept these facts as data that our current understanding just isn't there yet?
(and that the second best option, for now, is to physically isolate these systems during testing and training)
One argument is that it is difficult to replicate the entire internet, or even just a significant portion of it, realistically enough to fool agents that they are in the real-world.
But conversely, I think funny enough this might work, if only because at least a few of the recent 'AI escapes containment during testing' incidents appear to have involved agents who broke out and subsequently continued working without ever stopping to marvel at just how realistic this 'fake internet' looked.
The main hurdle I guess, is just that doing something like this would be very expensive, tedious, and, if the agents are sufficiently smart, possibly useless in the end. But all this money has to be good for something - and I can't help but feel like spending money to make sure you don't die, so you can live long enough to spend the rest of it, should be as good a motivation as any.
Not saying it's a bad idea, but to "just" airgap the servers...how do developers setup run and debug tests and look at the results? Do you fly the developers to the data centers and make them work there? Do you let developers bring a cell phone so they can ask a colleague when something out of their domain goes wrong? I think it would become very tempting to delegate to AIs to administer and monitor everything.
EDIT the way the US government does this is by running separate air-gapped networks (JWICS, NSANet), with their own fiber connecting datacenters to secured buildings. This would take time to build, so if you think you'll want it in the future you should start now.
I think you just answered your own question. It's not just money. It's that implementing proper safeguards is slow and tedious. It adds a lot of friction to development and testing. And these labs are screaming ahead at 1000mph in the greatest commercial arms race in human history. Since they are essentially unregulated, they put in place whatever safeguards they want to put in place, and so of course they're going to cut corners. This is why we need new regulatory agencies and laws to force them to comply, though the odds of that happening in the current environment seem very low. There is a lot of chatter about the risks, and the Overton window has shifted, but in the US the issue is already politicized along party lines, nearly ensuring gridlock on any substantive new policies.
Whenever Google finally implements a Dark Mode feature in Google Docs to stop blinding their users anytime they want to use Google's services after 11 PM at night with the lights off in bed
I think there is a:
But if Google can't do this by 2031, then I think I would expect like a ~95% chance that it will just never happen.[1]
Specifically, because as time passes, it becomes overwhelmingly more likely that Google will just sunset another widely-used, popular consumer product like Google Docs before ever giving it a quality of life feature.