Inspired by Daniel and Romeo https://x.com/DKokotajlo/status/1986231876852527270, I'm posting a memo on Takeoff, which I plan to expand into a proper article.

This memo comes from notes I took from a talk from Daniel, which have been synthesized by Claude for breviy:

Key Predictions

• Transformative AI timeline: Late 2028
• Key prediction: All existing benchmarks solved by 2030
• p(doom): 10%

Current Gaps

1. Reasoning capabilities (mostly solved)
2. Agency and long-horizon skills:
   • Error correction
   • Goal pursuit
   • Hierarchical planning
3. Data-efficient learning

Proposed Solutions

1. New models and architectures (e.g., recurrence)
2. Language Model Programs (LMP)
3. Scaffolding systems
4. AI bureaucracies
5. Increased inference compute
6. "Just train it to have those skills" approach

Stages of AI Development

The RE→RS→G→ASI Framework

1. Research Engineer for AI Research (we are entering here with Kosmos)
   • "Devin actually works"
   • Significant speedup in AI research
2. Research Scientist
   • "CEO talks to the cluster"
   • Autonomous white paper production
3. Genius
   • Qualitatively better than human researchers
4. Artificial Superintelligence
   • Capable of anything
   • Equivalent to 100k human teams

Technical Specifications:

Compute Requirements

• 10^14 text predictions/rollouts on agentic paths
• 10^23 floating point operations for transformative capabilities
• GPT-4 pre-training: ~2.15 × 10^25 FLOPS for comparison

Time Horizons

• t-AGI (10 second AGI to 10 day AGI)
• Year-AGI scaling
• Annual 1 OOM (order of magnitude) improvements expected

Critical Questions and Concerns

On Full Automation

1. Robotics gap: "If TAI is Industrial Revolution level, what about physical automation?"
2. Job stickiness: Institutional and social lags in job displacement
3. "Wet robots": Humans as interim solution

The Capitalism-AI Nexus

• "Paperclips are a metaphor for money"
• "Capitalism is the ultimate Turing test"
• AI systems are "incentivized to be constrained by human values"

Societal Predictions

Early prediction market odds:

1. 1/3 Futurama scenario
2. 1/4 Fizzle out
3. 1/5 Dystopia

Risk Scenarios

• Cyberpunk 2077-style hacking risks
• 43% of Americans believe civil war is somewhat likely
• Democracy at its peak by voter count (2024)
• Unprecedented political deepfakes
• Cultural homogenization
• Human disempowerment (sci-fi scenarios)
• Potential for singleton control
• "OpenAI coming for everybody's jobs is a solace" -

Basic Critiques

• Logarithmic growth in atoms, exponential growth in bits
• "Folk theorem - bigger (model) is better"
• "We're very different from current machines"
• "Brains are way more complex than even the most sophisticated NNs"
• Adaptation and time remain key differentiators

Alternative Approaches

• Virtual societies and digital twins
• AI safety as cancer immune response
• Automated social science research
• a deceptively simple solution to alignment: "Train them to be honest"