The case for fine-grained tracking of compute for AI
TL;DR Current approaches to tracking AI compute primarily rely on a handful of hardware proxies (like FLOP/s and bandwidth) that primarily track GPU progress. These metrics are becoming less useful for accurately tracking compute for AI because they (1) measure theoretical ceilings rather than actual performance, (2) as architectures diversify...
May 1334