The part that stands out to me is the tradeoff between speed and security. If AI systems can do infrastructure work much faster than humans, there will be a strong temptation to let them modify lower-level systems too, even when those systems are security-critical. I think keeping the AIs separated from the underlying infrastructure for as long as possible makes sense. The more they can treat the cluster as a controlled service rather than having access to schedulers, kernels, networking and hypervisors, the smaller the attack surface. The difficult part will be resisting the pressure to give up those boundaries when performance gains become significant.
Interesting read. The biggest concern for me isn’t really whether the technology is possible, but how difficult it would be to defend against large numbers of cheap autonomous drones at the same time. Countermeasures can work well against individual drones, but scale and cost could become the real problem as well as the internet and satellite signal communications. 😙
The part that stands out to me is the tradeoff between speed and security. If AI systems can do infrastructure work much faster than humans, there will be a strong temptation to let them modify lower-level systems too, even when those systems are security-critical. I think keeping the AIs separated from the underlying infrastructure for as long as possible makes sense. The more they can treat the cluster as a controlled service rather than having access to schedulers, kernels, networking and hypervisors, the smaller the attack surface. The difficult part will be resisting the pressure to give up those boundaries when performance gains become significant.