Josh You

If we ignore misalignment risk, why are you envisioning a world where ASI acts like a benevolent dictator, such that it is up the ASI's own values and decisions whether the middle powers have significant influence over the world? Rather than human customers and users of ASI-powered products and services around the world creating a balance of power (akin to how AI works today)? I think alignment should include be "don't be a dictator", not just "don't be an evil dictator".

Very cool research overall!

America has over 3,000 utilities, not one. There is no authority in the US which can order "disconnect these 50 lines in these 12 states" in a 20 minute window. 

This doesn't sound right to me. The relevant level of institution here is grid operator, not utility, and eight grid operators cover most of the US. Either way, any electrical grid has an institution that manages the grid in real time and responds to incidents like shortages and natural disasters. I don't see how the NZ plan as described falls outside the standard toolkit... (read more)

Update: the internal/external distinction only matters for the F-35. The F-15E and F-18 Super Hornet don't have internal bomb compartments. I think those three are the main fighter-bombers being used by the US and Israel in this campaign.

A modern fighter-bomber can carry 7-15 tons of ordinance, including external attachments (with penalties for aerodynamics and radar signature). Do we know how much US and Israeli fighters have been carrying per sortie on average? 

edit: One F-35 variant has an internal capacity of three tons and a total capacity of nine tons.

:re "Companies leaving without the company" it's worth thinking about what the equilibrium is. Foreign companies in currently-allied countries (e.g. Mistral which is founded and heavily staffed by Meta and GDM alumni) are currently poaching US lab staff or buying US chips in moderately large quantities with either no objection or the support of the US government. For the US government to countenance changes to chip policies or sanctions(!) with respect to such companies, the political and technological situation would have to change significantly from today.

Don't know how to find comprehensive data but they had a development pipeline of 1100 homes in 2022 against a population of 13k. So that might increase their population by ~15%. And about 500 units approved in the pipeline right now. I think the original 1100 pipeline includes the now-opened "Emery" development which had 500 units.

If they approved housing at the rate of Seattle, the leader among large US cities, they would be approving around 1500 per decade. So it seems fast, though at a population density of 10k/sq mile they still might take a couple dec... (read more)

Isn't Emeryville kind of doing this? Though I'm not sure if they're maxing out the envelope of housing production from real costs even if a city government goes 100% YIMBY.

Bit of feedback: would be helpful if you explicitly stated your estimated number of H200 and Huawei chips and/or provide a B300-eq conversion table, so they are more comparable to other reports that are quoted just in number of chips. I understand how you do the conversion but it is not super apparent in the post.

One thing I don't know is when data center investments get committed to specific customers. Google and Amazon are Anthropic's two main compute partners and will spend $200B each in capex this year and are presumably planning and developing sites for many more hundreds of billions by 2028. So one possible view of it is that their capex creates a window, and Anthropic's eventual share depends on its funding and revenue. But Google and Amazon don't quite know how their 2027-2028 data centers will be allocated.

In general, for large data centers the specific la... (read more)

A good term for 10^20 FLOP would be useful. This would make modern models around 100k to 10 million of this unit, which is a tangible number. Some people, e.g. at DeepMind tried to make "petaflop-days" (8.64e19) a thing but it didn't catch on. 

Another point here is that elections are an additional check after the courts, Congress, etc. US presidential elections are not administered by the federal government, they are administered by the states. So to interfere with elections, the president can't just fill election boards with cronies or give orders to anyone in his chain of command to rig the election. He'd have to forcibly manipulate or interfere with state officials and state governments, risking direct conflict with states. And if he doesn't interfere with the election and the states announce... (read more)

In-practice most federal offices have deferred to what the Supreme Court says, but we haven’t really seen what happens when e.g. a sitting president insists on an interpretation of the constitution that disagrees, and the constitution itself provides no clear answer.

This is a somewhat confusing statement. To be clear, it's extremely common for the president to disagree with courts on the law or Constitution: this happens dozens of times per presidential term. And when they lose in court the president may declare that they still think they are right and the... (read more)

Yeah I think leading labs generally retrain their base models less often than every 6 months (but there's a lot we don't know for sure). And I believe this most likely has to do with a production AI model being the result of a lot of careful tuning on pre-training, mid-training, post-training etc. Swapping in a new base model might lead to a lot of post-training regressions that need to be fixed. And your old base model is a "lucky" one in some sense because it either was selected for doing well and/or it required lots experiments, derisking runs, etc. Eve... (read more)

you should be more uncertain about the METR benchmark's external validity than what these error bars show. 

but your baseline uncertainty about key facts about AI progress in general should also often span much more than one order of magnitude between your 2.5th percentile and 97.5th percentile guess. the METR results add a lot of value and I don't think these error bars are a big deal in the scheme of things.

Most successful startups slow down a lot after a brief hypergrowth phase. We should be looking for signs that AI companies like OpenAI and Anthropic* are experiencing unusually long and persistent hypergrowth: surprisingly little slowdown in growth, or maintaining >2x growth/year at surprisingly high revenue levels like 100B. They are both already growing very surprisingly fast for companies with multiple billions in revenue, to be clear, but whether that continues is valuable evidence.

This could be a sign that present-day models have a higher economic ... (read more)

I'd flip it around and ask whether Gabriel thinks the best models from 6, 12, or 18 months ago could be performing at today's level with maximum elicitation.

I think the linked tweet is possibly just misinterpreting what the authors meant by "transistor operations"?  My reading is that "1000" binds to "operations"; the actual number of transistors in each operation is unspecified. That's how they get the 10,000x number - if a CPU runs at 1 GHz, neurons run at 100 Hz, then even if it takes 1000 clock cycles to do the work of neuron, the CPU can still do it 10,000x faster.

Hmm I see it. I thought it was making a distinct argument from the one Ege was responding to here, but if you're right it's the same one.

T... (read more)

OpenAI can probably achieve Meta/Google-style revenue just from monetizing free users, since they're already one of the biggest platforms in the world, with a clear path to increasing eyeballs through model progress+new modalities and use cases, and building up an app ecosystem (e.g. their widely rumored web browser). An anonymous OpenAI investor explains the basic logic:

The investor argues that the math for investing at the $500 billion valuation is straightforward: Hypothetically, if ChatGPT hits 2 billion users and monetizes at $5 per user per month—“ha

Google DeepMind uses Nvidia very sparingly if at all. AlphaFold 3 was trained using A100s but that's the only recent use of Nvidia by GDM I've heard of. I think Google proper, outside GDM, primarily uses TPUs over GPUs for internal workloads, but I'm less sure about that.

Google does buy a lot of Nvidia chips for its cloud division, to rent out to other companies.

• xAI and Meta still use Nvidia. Almost every non-frontier-lab, non-Chinese AI chip consumer uses Nvidia
• And Alphabet, Amazon, and Broadcom, the companies that design TPU and Trainium, have the 4th, 5th, and 7th biggest market caps in the world. 

I think it's possible that the market is underpricing how big a deal Anthropic and Google DeepMind, and other frontier labs that might follow in their footsteps, are for overall AI chip demand. But it's not super obvious.