roottwo

I think this is a good question. I'd love to hear from people with experience building frontier models have to say about it.

Meanwhile, my first pass at decomposing "activities that go into creating better models" into some distinct components that might be relevant in this discussion:

1. Core algorithmic R&D: choose research questions, design & execute experiments, interpret findings
2. ML engineering: build & maintain distributed training setup, along with the infra and dev ops that go along with a complex software system
3. Data acquisition and cura

... (read more)