This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I suspect that this fails at the part where we actually make sure that the spec/formalization reflects what we think it does in natural language/human concepts (though, the exact setup found in the linked tweet is unusually clean/clever).
Off the top of my head, some possible failure modes I can imagine with the quoted scheme are:
Somehow the agent can write code that reproduces the honeypot's contents for a very large, practically relevant set of honeypots.
Somehow the agent is able to achieve things by means other than writing code. (This is a special c... (read more)