Epistemic Status: Quick tests, most of this was done in less than 48 hours

TL;DR: Can you skimp on GPU VRAM?  8bit quantized seems fine, for 4bit it depends. 

I was asked by @Teun van der Weij, to what degree one can run evaluations on quantized models, and I was unsure. I have run some evaluations with Llama 3 and have some quick comparisons now. 

Main Quantization Schemes

Here is a list of some different quantization schemes discussed:

• GGUF - Special file format used in Llama.cpp. Not supported in transformers.
• BNB - BitsAndBytes, the original default in huggingface transformers.
• BNB NF4 - Alternative mode for bits and bytes, "4-bit NormalFloat"
• HQQ - Half-Quadratic Quantization, supports 1-8 bits.
• GPTQ - One of the older quantization methods. Requires training data
• AWQ - "Activation-aware Weight Quantization". Requires training data

Llama 2

Some previous papers have compare perplexity of different methods. We can see an example of some research shown in the recent research paper using HQQ quantization:

Table 1: Comparison of perplexities at different levels of quantization with different methods on WikiText2 dataset for Llama 2 7b, 13b and 70b.

However, it is quite unclear what effect this has on real world performance. 

Llama 3 8B

Here is some examples of what perplexity looks like with different levels of quantization on Llama 3 8B, found from the llama.cpp repository^[1]. This only includes GGUF, and not any other quantization methods as far as I can tell.

Table 2: Comparison of perplexities at different levels of GGUF quantization on the WikiText2 dataset for Llama 3 8b.

This is all fine and good, but a lot of use are trying to do interpretability and whatnot, and I personally have found this easiest when using the HuggingFace transformers library. While for some it may make sense to retool, what about those of us who do not want to? While we can wait for potential compatibility with GGUF, there are other quantization methods we can use in the meantime, and I have tried to run some tests.

Llama 3 8B Instruct

Here are some benchmarks with Llama 3 8B, run with different quantization levels and schemes on MMLU, WMDP (% accuracy), and The Pile (perplexity):

gptq4 - Acc=55.21% PPL=8.575 T=233s
awq4 - Acc=55.55% PPL=8.483 T=270s

We can see that the scored going from 16-bit to 8-bit are relatively unaffected, so likely it is fine to use 8bit. Going to 4-bit has a more noticeable effect, but it is not massively different on a qualitative level. 

Chain of Though

OK, sure, maybe single-token prediction tasks are relatively unaffected, but Chain of Thought (CoT) requires many "correct" tokens at a time. Maybe these are affected more? I run some experiments on Minerva MATH Algebra dataset, with zero-shot and Chain-of-Thought. (I only use only a subset "Algebra" because CoT takes a long time to generate). Here are the results:

Overall, we can see again that for 8-bit quantization, the effect on performance doesn't seem that large, though there is some noticeable degradation when going to 4-bits.

** One thing to note is that the 8-bit implementations, for some reason, were rather slow. I think this is likely some problem I have with torch.compile / torch._dynamo (since the effects were not as noticeable for 4bits or for perplexity results) but I did not have time to test this. This may be because in some runtimes, the weights need to be de-quantized at every step. Also, these results were run on an (architecturally) older A4000 which does not support FP8/FP4 compute, so results may vary.

Conclusion

• It seems like it is basically fine to use fp16 models or int8 quantized models relatively interchangeably.
• You may be able to get away with using 4-bit quantized models depending on the task, but there is some noticeable degradation.

1. ^{^}

   Note that some earlier implementations of GGUF with Llama 3 8B had some error when loading the tokeniser, and had much worse performance because of this.

2. ^{^}

   Note that float16 and bfloat16 experiments were run on a dual-GPU setup, so the time may not be directly comparable