The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD

Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both

Agreed! Relatedly: https://news.ycombinator.com/item?id=49599953

How did I miss zstd?

Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.

    gzip  7.3%  21s  9s
    bzip2 4.6% 251s 50s
    bzip3 3.3%  82s 69s
    zstd  6.9%   2s  3s
    lzma  4.7%  51s  3s

At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.

But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.

zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)

zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different

Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression.

Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.

Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.

After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.

all hail zstd, the one format to rule them all

Or lz4

Not really, it's a popular dictionary-based compression format.

I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things.

Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?

The only reason to pick anything but zstd is that platform support might be better.

If your platform/sdk/browser/standard-library comes with zlib/gzip/.. then it's often easier to just pick that.

No new dependencies is always a win. App size. Security, etc.

Otherwise, if zstd is easy to add, IMO I would always prefer, zstd, lz4 or brotli.

zstd or lz4

”Not really" what?

It's hard to understand what point you're trying to make. Can you clarify?

It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.

I think you're being a bit pedantic.

A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything.

Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.