LESSWRONG
LW

Computer ScienceComputeWorld Optimization
Frontpage

70

Avoid the abbreviation "FLOPs" – use "FLOP" or "FLOP/s" instead

by Daniel_Eth
10th Jul 2022
1 min read
13

70

Computer ScienceComputeWorld Optimization
Frontpage

70

Avoid the abbreviation "FLOPs" – use "FLOP" or "FLOP/s" instead
12paulfchristiano
7qwertyasdef
5paulfchristiano
3Daniel_Eth
3hold_my_fish
9jefftk
20Steven Byrnes
9Daniel_Eth
5jefftk
5lennart
2smcurtis
2zeshen
0ponkaloupe
New Comment
13 comments, sorted by
top scoring
Click to highlight new comments since: Today at 3:45 PM
[-]paulfchristiano3y125

"Floating point operations per second" is usually abbreviated FLOPS. I think using FLOPs as a synonym for FLOPS seems wrong in a particularly confusing way. It really suggests that FLOP is the acronym and the "s" is for a plural. Compare to someone saying SMs instead of SMS.

That said, I agree that using FLOPs as the plural of "floating point operation" is confusing since it is pretty similar to FLOPS. It's also not something I've seen much outside of the futurist crowd (though I am guilty of using it fairly often). Another way in which the ML and futurist usage is nonstandard is that we often use FLOPS sloppily to talk about very low precision or even integer arithmetic.

I haven't personally seen this cause very much confusion, in my experience it maybe increases the amount of "compute stock vs flow" confusion by like 25-50% from a small-ish baseline? I suspect it's small enough that accessibility and using standard notation is significantly more important. (Though again, my sense is that FLOPs for "floating point operations" is not really standard and not particularly accessible, so it seems more plausible futurists should stop doing that.)

Not sure what's realistic to do here. I'm a bit inclined to use "operations" or "ops" to both avoid this ambiguity and be more inclusive of integer arithmetic, but I kind of wouldn't expect that to stick and maybe it's bad in some other way. Using FLOP as a plural noun is pretty unsatisfying and I suspect it is unlikely to stick for fundamental linguistic reasons. Switching to using FLOP/s instead of FLOPS may be realistic, though I think flop/s is a bit less weird-looking to me.

ETA: though I think openphil has been using FLOP/s and FLOP as you suggest in their writing, so maybe I'm wrong about whether it's realistic. I think the draw to add an s to plurals is strongest in spoken language, though unfortunately that's also where the ambiguity is worst.

Reply
[-]qwertyasdef3y71

As for using FLOP as a plural noun, that's how other units work. We use 5 m for 5 meters, 5 s for 5 seconds, 5 V for 5 volts, etc. so it's not that weird.

Reply
[-]paulfchristiano3y52

This feels easier to me when writing and significantly harder for spoken language (where I have a strong inclination to add the s). I guess it's probably worth thinking separately about talking vs writing, and this post is mostly about writing, in which case I'm probably sold.

Reply
[-]Daniel_Eth3y30

In spoken language, you could expand the terms to "floating-point operations" vs "floating-point operations per second" (or just "operations (per second)" if that felt more apt)

Reply
[-]hold_my_fish3y3-9

Not sure what's realistic to do here.

Could write "FLOPS-seconds" maybe, which, although a bit wordy, resembles the common usage of "kilowatt-hours" instead of a more directly joule-based unit.

Reply
[-]jefftk3y90

Can you point to example usages that are causing confusion?

Reply
[-]Steven Byrnes3y2013

Here's a quote from someone on EleutherAI discord 2 weeks ago:

Does anyone know how much training compute would be needed for AGI? The human brain uses about 10^16 flops and GPT-3 required about 10^23 flops for training which is 10 million times more than the human brain. It seems like we have the compute needed for AGI today but we still don't have AGI.

I believe the first "flops" is FLOP/s and the second "flops" is FLOP, although I'm not 100% confident. I replied asking for clarification and didn't get a straight answer.

Reply
[-]Daniel_Eth3y90

FWIW, I am ~100% confident that this is correct in terms of what they refer to. Typical estimates of the brain are that it uses ~10^15 FLOP/s (give or take a few OOM) and the fastest supercomputer in the world uses ~10^18 FLOP/s when at maximum (so there's no way GPT-3 was trained on 10^23 FLOP/s).

If we assume the exact numbers here are correct, then the actual conclusion is that GPT-3 was trained on the amount of compute the brain uses in 10 million seconds, or around 100 days. 

Reply
[-]jefftk3y50

That's a great example, thanks!

(I'm used to "flops" always being used to describe the capacity of machines (as in, per second) and not total computation as in this example.)

Reply
[-]lennart3y54

Agree that this discussion is surprisingly often confusing and people use the terms interchangeably. Unfortunately, readers often referred to our training compute measurement as a measure of performance, rather than a quantity of executed operations. However, I don't think that this is necessarily due to the abbreviations but also due to the lack of understanding of what one measures. Next to making the distinction more clear with the terms, one should probably also explain it more and use terms such as quantity and performance.

For my research, I've been trying to be consistent with FLOPs (smaller case s) referring to the quantity. While FLOPS or FLOP/s refer to the performance: operations per second. (FWIW, during my time in computer engineering, it has been the norm to use FLOPs for quantity and FLOPS for performance.)

The term Petaflop/s-days also helps - outlining how long the performance (Petaflop/s) runs for how many days, therefore measuring a quantity of operations.

Note it gets even more complicated once we take the number representation (floating point 32bit, 16bit, or even bfloat16) into consideration. Therefore, I'm also in favor of maybe switching at one point towards OPs and OP/s and also document the used number representation for actual technical documentation (such as reporting the compute of ML models).

Reply
[-]smcurtis2y20

Note that @lennart has since recommended using FLOP for quantity and FLOP/s (or FLOPS) for performance.

Reply
[-]zeshen3y20

This seems like a good idea, although I feel that the number of people who use FLOPS to mean FLOP per second are still going to grow faster than those who switch to using FLOP/s, as most introductory material from a simple google search still refer to FLOPS as FLOP per second. Perhaps an edit to the wiki page could expedite the switch? 

Also, will FLOP/s still be pronounced 'flops' or will it now be 'flop per second'?

Reply
[-]ponkaloupe3y00

i can’t think of another (pseudo-)acronym which gets used as an all-caps unit, off the top of my head. i may toy around with “Flop” as a unit, like GFlop for a billion operations, GFlop/s for a billion operations per second.

Reply
Moderation Log
Curated and popular this week
13Comments

Especially in discussions about AI, the abbreviation "FLOPs" is being used for both "floating point operations per second" (a measure of computational power) and "floating point operations" (a measure of total computations, and equivalent to the previous term times seconds). This is ambiguous and confusing. For clarity, I propose people avoid this specific abbreviation and instead use the alternatives of "FLOP" (for floating point operations) and "FLOP/s" (for floating point operations per second).

Mentioned in
9Analysing a 2036 Takeover Scenario