Most features completed - #5
Conversation
…nches Bring rapidgzip-rust to decoder feature parity with C++ rapidgzip for indexes (GZIDX/gztool/BGZI), seek/prefetch, ranges, zlib/raw DEFLATE, streaming stdin with parallel spill, analyze, and CLI ergonomics. Add a fair comparative harness and a local synthetic benchmark snapshot vs rapidgzip 0.15.2.
…ckend Ship feature-complete rapidgzip-core and CLI beyond the incomplete crates.io 0.1.0 stub: parallel gzip/BGZF/zlib/raw paths, indexes and seek, decode_read spill, performance work, InflateBackend consolidation, fair C++ 0.16 harness, CHANGELOG and release tooling. Bumps workspace version to 0.2.0.
Add optional `isal` feature (IsalInflater as ActiveInflater) for faster P=1 inflate when libisal is available; default builds stay zlib-rs. Map ISA-L INPUT_DONE/FINISH and prefetch refunds to zlib-compatible consume/STREAM_END. Update docs, fair re-bench snapshot vs C++ 0.16 ISA-L (~1.35× P=1 geo mean).
Apply rustfmt so workspace fmt --check and release gates pass cleanly.
Defer RawInflater allocation in IsalInflater until Block flush or non-empty prime/dictionary so pure NoFlush P=1 stays ISA-L-only. Add Ubuntu CI job installing libisal-dev and testing --features isal. Refresh crate/backend docs and PERFORMANCE_AUDIT residual for shipped isal.
Reject multi-element raw_crc32_list at DecoderBuilder::build (single whole-stream CRC only). Stash isal prime/dictionary until first Block so NoFlush seek resume never allocates zlib-rs. Update monomorphization docs and CI isal tests to --all-targets.
Drive InflateFlush::Finish through repeated isal_inflate until STREAM_END (or budget/error), so BGZF one-shots complete when tmp_out drains after INPUT_DONE. Label CLI as version+isal and matrix rows as rapidgzip-rust-isal when libisal is linked. Document Unreleased changes and bench fairness.
Skip full inflate_state zero on IsalInflater create (~87 KiB); use ISA-L crc32_gzip_refl for verify when the isal feature is on; run BGZF with one worker inline without channel reordering. Profile notes: P=1 isal already beats zlib-rs and C++ on fair corpora; opts add a few percent.
ISA-L Finish multi-step drain, lazy Block/pending prime-dict, create/CRC optimizations, single-thread BGZF inline path, raw_crc32 multi-value reject, fair-bench isal labeling, and CI isal job. See CHANGELOG.
|
Thank you @hilather for the substantial amount of work and thought that went into this pull request. It provided a valuable map of the broader feature surface and helped motivate several focused designs. Since this PR was opened, its major capabilities have landed through smaller, reviewable replacements based on the evolving
Those replacements also preserve The optional ISA-L backend was evaluated separately in #10 and deliberately not adopted because it adds a native build and maintenance burden without a demonstrated performance advantage over zlib-rs on the measured workloads. Automatic tempfile spooling and disabled checksum verification also remain intentional non-goals for the default APIs. Three narrower ideas remain worth evaluating independently: observable indexed-reader caching/prefetch, broader decoded-buffer recycling, and expanded fair-benchmark tooling. I am opening focused draft PRs for those and will cite this PR in each one. I am therefore closing this PR as superseded rather than merging its now-conflicting aggregate branch. Thank you again—the ideas here materially influenced the project even though the implementation ultimately landed in a different form. |
|
The three focused follow-ups described above are now open as independent draft PRs based on current
Each cites this PR as its origin and contains a critical design assessment, explicit scope/non-goals, an implementation plan, and measurement/acceptance gates. |
3 participants
Hi,
Came across your project while working on another problem where I needed a feature complete version of rapidgzip in rust. Finished it with vibe coding. Thought I'd let you know with this pull request in case you this helps you.