[POC] zstd decompression

[rkistner]

This investigates the possibility of decompressing zstd data in the core extension, which could allow us using zstd data in the protocol. This POC only looks at the zstd decompression itself and tests its performance, and does not actively use it anywhere.

Usage:

cargo build -p powersync_loadable --release
sqlite3 test.db # db with compressed data pre-loaded
.load ./target/release/libpowersync
with dictionary as materialized (select readfile('dictionary') as dict)
select sum(length(zstd_decompress_text(data, dict))) from compressed_data, dictionary;

On my machine, this takes around 500ms to decompress 80MB of data over 100k rows. This will likely be more efficient if we parse the dictionary up-front.

This increases linux release build size from around 400kb -> 487kb. Adding compression support would add another 260kb or so, and I don't think we have a good use case for that on the client.

To actually use this with compressed data, we'd need to additionally:

1. Use streaming decompression instead of pre-allocating buffers (I think this works by decompressing a block at a time).
2. Manage dictionaries - we'd need to persist them somewhere, then load, parse and cache them in memory when used.
3. Implement changes in the protocol and service to send compressed data and dictionaries to the client.

[simolus3]

This will likely be more efficient if we parse the dictionary up-front.

Would we re-use the dictionary across sync lines? Is the dictionary dynamic and sent by the service?

[rkistner]

Would we re-use the dictionary across sync lines? Is the dictionary dynamic and sent by the service?

In short, yes. My initial idea is that compression would be used for the data, rather than the protocol. This allows keeping the data compressed as is from bucket storage -> sync stream (websocket) -> ps_oplog, potentially even in the data tables (data tables would be opt-in, since this would slow down queries).

However, if we do compression on the data of individual rows, the compression ratio for small rows isn't great (around 0.6 of the original size in my tests). However, if we pre-train a dictionary on the same data, the compression ratio can be as good as 0.2 or 0.1, a 5-10x reduction in size.

This does have some implications:

1. We'll need to train a separate compression dictionary (or multiple) for each bucket, to avoid leaking data between buckets/users.
2. We'll need a way to indicate which dictionary is used for which bucket or operations.
3. We'll need a way to get those dictionaries to the client.
4. We'll need to manage those dictionaries on the client.

So overall the project becomes quite complex, but I don't think compression is worth it without using dictionaries.

This PR is just one small piece of evaluating the feasibility of the project: Can we efficiently do decompression on the client?