[Target audience: Me from a week ago, and people who have some understanding of ML but want to understand transformers better on a technical level.]

Free advice for people learning new skills: ask yourself random questions. In answering them, you’ll strengthen your understanding and find out what you really understand and what’s actually useful. And some day, if you ask yourself a question that no one has asked before, that’s a publication waiting to happen!

So as I was reading up on transformers, I got fixated on this question: where are the 175 billion parameters in the architecture? Not in the literal sense (the parameters are in the computer), but how are they “spent” between various parts of the architecture - the attention heads vs feed-forward networks, for instance. And how can one calculate the number of parameters from the architecture’s “size hyperparameters” like dimensionality and number of layers?

The goal of this post is to answer those questions, and make sense of this nice table from the GPT-3 paper, deriving the $n_{p a r a m s}$ column from the other columns.

Most of Table 2.1 from the GPT-3 paper. How can we derive the number of parameters from the “size hyperparameters” (rightmost columns)?

Primary Sources

Lots of resources about transformers conjure information from thin air, and I want to avoid that, so I’m showing all my work here. These are the relevant parts of the sources^[1] we'll draw from:

Exhibit A: Equation 2 from the GPT-1 paper.

Exhibit B: Half of Figure 1 from the original transformer paper.

Exhibit C: An unnamed equation on page 5 in the original transformer paper. In the terminology of GPT, $d_{k}$ = $d_{v}$ = $d_{h e a d}$ , and $h = n_{h e a d s}$

Exhibit D: Equation 2 in the original transformer paper.

Three more details we’ll use, all from Section 2.1 of the GPT-3 paper:

The vocabulary size is $[n_{v o c a b} =] 50257$ tokens (via a reference to Section 2.3 of the GPT-2 paper)
The feed-forward networks are all a single layer which is “four times the size of the bottleneck layer”, so $d_{f f} = 4 d_{m o d e l}$
“All models use a context window of $n_{c t x} = 2048$ tokens.”

Variable abbreviations

I’ll use shorthand for the model size variables to increase legibility:

$\begin{matrix} n_{l a y e r s} & = & x d_{m o d e l} & = & y n_{h e a d s} & = & z d_{h e a d} & = & w n_{v o c a b} & = & v n_{c t x} & = & u \end{matrix}$

Where are the Parameters?

From Exhibit A, we can see that the original 1-hot encoding of tokens U is first converted to the initial “residual stream” $h_{0}$ ...

michael wan

michael wan

michael wan

michael wan

michael wan

michael wan