I think this is a good question. I'd love to hear from people with experience building frontier models have to say about it.
Meanwhile, my first pass at decomposing "activities that go into creating better models" into some distinct components that might be relevant in this discussion:
I think this is a good question. I'd love to hear from people with experience building frontier models have to say about it.
Meanwhile, my first pass at decomposing "activities that go into creating better models" into some distinct components that might be relevant in this discussion:
- Core algorithmic R&D: choose research questions, design & execute experiments, interpret findings
- ML engineering: build & maintain distributed training setup, along with the infra and dev ops that go along with a complex software system
- Data acquisition and curation: collect, filter, clean datasets; hire humans to produce/QA; generate synthetic data
- Safety research and evaluation: red-teaming, interpretability, safety-specific evals, AI-assisted oversight, etc.
- External productization: product UX and design, UX-driven performance optimization,
... (read more)