I noticed that some AI-safety-focused people are very active users of coding agents, often letting them run completely unrestricted. I believe this is a bad standard to have.

I do not think that running Claude Code itself is catastrophic, but risk profiles are not binary and I believe that this new interface should be used more cautiously.

I will explain my thoughts by going over the standard reasons someone might give for --dangerously-skipping-permissions:

#1 It's not dangerous yet

Many people are using Claude Code, Codex, Cursor, etc. and there haven't been any catastrophic accidents yet, so it is a reasonable claim.

Even if today's models are safe, I expect they will get more dangerous down the line. The important thing is knowing where your red lines are. If the models get slightly better every two months, it is easy to get frog-boiled into riskier behavior by not changing your existing habits.

If you are fine with the small chance of all your files being deleted, that's okay; just define this explicitly as an acceptable risk. A year from now the consequences of a misbehaving agent could be worse and you might not notice.

#2 The benefits outweigh the risks

Modern-day agents are very, very useful, so you should consider their advantages for your productivity, especially if your work is important.

I think this is true, and I also think that agents are only going to get more useful from this point on. If your current workflow is autonomously running simulations and building new features, your future workflow might be autonomously writing complete research papers or building dozens of products. This will feel amazing, and be very effective, and could also help the world a lot in many important ways.

The reason people are worried about super-intelligence is not because it won't contribute to personal productivity.

The risks and benefits of these powerful, general-purpose tools both arise from the same set of capabilities, so a simple utilitarian calculus is hard to apply here. The benefits are concrete and quantifiable and the risks are invisible - they always will be. Instead of making sure it's more beneficial than harmful, I believe that when working with agents, it is better to set up hard limits on the blast radius, then maximize your productivity within these limits.

#3 If anyone builds it

There is the MIRI viewpoint of a big, discrete leap in cap...

doctorfarhan

doctorfarhan

doctorfarhan

doctorfarhan