At work we've recently been using zstd as a better-compressing alternative to gzip, and overall I've been pretty happy with it. A minor documentation gripe, though, is that the behavior around multithreaded compression is a bit unclear. I understand it's chunking the work and sending chunks to different threads to parallelize the compression process, and this means that I should expect to see better use of threads on larger files because there are more chunks to spread around, but what is the relationship?

When I look in man zstd I see that you can set -B<num> to specify the size of the chunks, and it's documented as "generally 4 * windowSize". Except the documentation doesn't say how windowSize is set.

From a bit of poking at the source, it looks to me like the way this works is that windowSize is 2**windowLog, and windowLog depends on your compression level. If I know I'm doing zstd -15, though, how does compressionLevel=15 translate into a value for windowLog? There's a table in lib/compress/clevels.h which covers inputs >256KB:

See the source if you're interested in other sizes.

So it looks like windowSize is:

• ≤1: 524k
• 2: 1M
• 3-8 (default): 2M
• 9-16: 4M
• 17-19: 8M
• 20: 32M
• 21: 64M
• 22: 128M

Probably best not to rely on any of this, but it's good to know what zstd -<level> is doing by default!

Comment via: facebook, mastodon, bluesky