OpenAI Jalapeno suggests that OpenAI is serious about inference in FP4 for 40T-60T total param models of 2027. Its scale-up rack holds 128 chips, each with a single compute die and 6 stacks of 12-Hi HBM4 (3 GB per DRAM die, 2.5 TB/s per stack, 14 ms to fully read). Jalapeno's FLOP/s are mostly FP4, with an FP4/FP8 ratio of 3.9, unlike Nvidia Rubin and AMD MI455X that have an FP4/FP8 ratio of 2. So just as Rubin demonstrates Nvidia's conviction that training only needs FP8 (and not BF16) by using an FP8/BF16 ratio of 4.4 (unlike the usual 2), similarly Jalapeno demonstrates OpenAI's conviction that inference (in FFNs) only needs FP4 (and not FP8), even for the 50T total param class models (the FP8 FLOP/s are likely for attention and such, and BF16 is absent entirely in Jalapeno).

Like Rubin and MI455X, Jalapeno uses 12-Hi HBM4 (14 ms to fully read), unlike this year's GB300 with its much slower 12-Hi HBM3E (36 ms to fully read). This makes it feasible to generate tokens with pipelines that have 3-4 stages (3-4 racks), yielding a time per output token similar to that of GB300 without pipelining (1 rack). Model params can then occupy 2-3 half-racks of HBM capacity (for the 3-4 stage pipelines), compared to just a single half-rack of HBM capacity for GB300 Oberon. For Jalapeno (27.6 TB of HBM4 per rack), that's 55T-83T total params in FP4, which now seems plausible for the 2027 models (at 3-4T active params). But since the models of 2027 have to also run on Rubin (20.7 TB per rack), the number of total params has to be 41T-62T instead (still using FP4).

The main update is a bet on FP4 for inference, which wasn't at all clear (for the biggest models) given Rubin and MI455X, since those chips also need to be good at pretraining (and so can't drop FP8). Since OpenAI is going this route, Anthropic might be doing the same.