Pluggable inflate backends, with ISA-L behind a feature by BenjaminDEMAILLE · Pull Request #10 · COMBINE-lab/rapidgzip-rust

BenjaminDEMAILLE · 2026-08-01T19:07:51Z

Sub-project 3 of 4 replacing #5. Stacked on #9, which stacks on #8. GitHub cannot base a cross-fork pull request on a fork branch, so this targets main and its diff contains both parents; review the top three commits only (Route the sequential gzip loop..., Add an ISA-L raw-inflate backend..., Cover the ISA-L backend in CI, Measure the inflate backends...). Merge #8 and #9 first.

What this does

Raw inflate now sits behind one crate-internal trait, InflateBackend. A type alias, ActiveInflater, picks the implementation at compile time: zlib-rs by default, ISA-L under the new off-by-default isal feature.

Only the paths that decode a whole stream from its start go through the alias:

Path	Backend	Why
Sequential gzip members	pluggable	whole-stream inflate, no bit-level resume
Single-stream zlib and raw DEFLATE	pluggable	same
BGZF blocks	pluggable	each block is a complete stream
Estimated grid (marker/window)	zlib-rs	needs `inflatePrime` and `Z_BLOCK`
`IndexedReader`	zlib-rs	resumes at arbitrary bit offsets

The two on the right name RawInflater directly, so a backend that cannot express a bit-accurate resume is not reachable from them by construction rather than by convention. The trait carries no dictionary call for the same reason: every stream the pluggable paths inflate starts at its own beginning.

The default build is unchanged: same dependencies, same behaviour, and since the alias is a concrete type, the same generated code.

Measurement

benches/inflate_backend.rs decodes a 16 MiB semi-structured log corpus single-threaded through each backend, compared through criterion baselines.

On Apple M-series against Homebrew isa-l 2.32.1:

Path	zlib-rs	ISA-L	Change
sequential gzip	1.49 GiB/s	1.24 GiB/s	+20.3% time
raw DEFLATE	1.56 GiB/s	1.29 GiB/s	+20.2% time

ISA-L loses on both, consistently, p < 0.05, confidence intervals disjoint. That is the honest result on the hardware available here, and the feature stays off by default. The isal CI job runs the same two benchmarks on x86-64, where ISA-L's assembly decoder is strongest and where this project has no local machine to measure; read its logs for that number.

One benchmark note worth stating plainly: an earlier corpus of verbatim-repeating text measured 3.9 GiB/s and compared nothing, because both backends spend that workload copying a handful of very long matches at memory speed. The corpus now carries pseudo-random fields so the Huffman decoding is actually on the clock.

Two things ISA-L's interface forces

inflate_state embeds a 64 KiB scratch buffer, so it is boxed rather than held inline.

More importantly, ISA-L reads ahead into a bit buffer, so at the end of a stream it has consumed input the stream does not own. Whole bytes still in that buffer are given back, or the gzip footer and the next member would be read from the wrong offset. The concatenated-member and BGZF tests are what catch this.

Verification

Full suite green with --features isal, including the indexed-seek tests, which is the check that the bit-accurate paths still run on zlib-rs.
Full suite, clippy, fmt, and rustdoc green on the default build.
New CI job installs libisal-dev and runs clippy plus the suite with the feature on.

Documentation

The split is in ARCHITECTURE.md, the ISA-L unsafe argument in SAFETY.md, the measurement in PERFORMANCE_AUDIT.md, and how to enable and reproduce it in README.md and the crate docs. SAFETY.md also gets a stale path corrected: the zlib wrapper moved to inflate.rs.

🤖 Generated with Claude Code

Every entry point required a `ReadAt` source, so gzip arriving on a pipe could not be decoded at all. Path 1 only ever moves forward through the compressed bytes, so extract the operations it uses into an internal `InputCursor` trait and implement it for both the positional `SourceCursor` and a new forward-only `StreamCursor`. Member framing, footer verification, trailing-garbage detection, and the output limit are then literally the same code for a stream as for a file, rather than a second implementation that could drift. Adds `Decoder::decode_stream` and `Decoder::stream_reader`, mirroring `Decoder::decode` and `Decoder::reader`. `Decoder::open` routes a path that cannot be read positionally to the streaming driver, which turns a call that failed before into one that succeeds. The CLI accepts `-` for standard input and follows the same routing. The streaming runtime is configured with one worker so the telemetry reports the concurrency actually in use. A streaming reader does not join its coordinator on drop, because that coordinator can be parked in a read against a producer that never writes again. No new dependency and no new unsafe code. Refs COMBINE-lab#6 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every case is fed through a source that implements only `Read`, so none of them can silently fall back to positional reads the way a `Cursor` would, and each is compared against the positional decode of the same bytes. Covers single-member, concatenated and empty members, BGZF with its EOF member, a fully stored stream, truncation inside DEFLATE and inside the footer, a corrupt CRC32 and a corrupt ISIZE naming the offending member, trailing garbage, the output limit, telemetry, `Box<dyn Read + Send>` through paraseq, dropping a reader against a stalled producer, and a slow producer writing in small chunks. On unix it also decodes a real `cat` pipe and checks that `Decoder::open` routes a FIFO to the streaming path while leaving a regular file on a parallel one. Refs COMBINE-lab#6 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Records what streaming input supports, what it does not, and why: the same verification as a regular file because it is the same sequential code, but no parallelism because every parallel path needs positional reads. ARCHITECTURE.md explains that the length snapshot is absent for a stream rather than mutable, that end of input plus the existing trailing-garbage check is what makes it authoritative, and why a streaming reader does not join on drop. SAFETY.md is unchanged; this feature added no unsafe code. Refs COMBINE-lab#6 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Reading into an empty buffer tail returns `Ok(0)`, which the cursor would read as end of input and turn into silent truncation. Callers only refill a drained window so this is unreachable today, but the failure mode is bad enough to be worth closing. Refs COMBINE-lab#6 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

First of four sub-projects reimplementing PR COMBINE-lab#5 in reviewable pieces. Covers index construction as a by-product of decoding, the native, GZIDX, .gzi, and gztool on-disk formats, and a separate IndexedReader for random access. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Thirteen tasks from index types through the four on-disk formats, the IndexedReader, decode-path wiring, interop tests, and documentation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>