DEFLATE analysis and the --analyze report

.github/workflows/ci.yml

@@ -45,6 +45,54 @@ jobs:
       - run: rustc --version --verbose
       - run: cargo test --workspace --all-targets --locked
 
+  interop:
+    # Index format interoperability against the reference tools. These tests
+    # are ignored by default because they need the tools installed.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - name: Install bgzip
+        run: sudo apt-get update && sudo apt-get install -y tabix zlib1g-dev
+      - name: Install indexed_gzip and rapidgzip
+        run: python3 -m pip install --user indexed_gzip 'rapidgzip==0.16.0'
+      - name: Build gztool
+        run: |
+          git clone --depth 1 --branch v1.8.2 https://github.com/circulosmeos/gztool.git /tmp/gztool
+          cc -O2 -std=gnu11 -o /usr/local/bin/gztool /tmp/gztool/gztool.c -lz -lm
+      - run: bgzip --version && gztool -h | head -1 && python3 -c "import indexed_gzip" && rapidgzip --version
+      - run: cargo test -p rapidgzip-core --test index_interop --locked -- --ignored --nocapture
+      # The analyze report is meant to be drop-in for rapidgzip's, so the
+      # reference tool is what checks it.
+      - run: cargo test -p rapidgzip-rust-cli --test analyze_interop --locked -- --ignored --nocapture
+
+  isal:
+    # The optional ISA-L inflate backend. It links a system libisal, so the
+    # default jobs cannot cover it.
+    runs-on: ubuntu-latest
+    env:
+      # isal-sys looks for the library under this prefix. The linker finds it
+      # under the multiarch directory either way, but the variable is what
+      # stops the crate from building ISA-L from source.
+      ISAL_INSTALL_PREFIX: /usr
+    steps:
+      - uses: actions/checkout@v7
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: clippy
+      - uses: Swatinem/rust-cache@v2
+      - name: Install libisal
+        run: sudo apt-get update && sudo apt-get install -y libisal-dev
+      - run: cargo clippy -p rapidgzip-core --all-targets --features isal --locked -- -D warnings
+      - run: cargo test -p rapidgzip-core --all-targets --features isal --locked
+      # The backend is a compile-time choice, so comparing means two runs. This
+      # is the only x86-64 measurement available to the project, which develops
+      # on AArch64, and it is where ISA-L is supposed to win. Shared runners are
+      # noisy, so read the sign and rough size, not the exact percentage.
+      - run: cargo bench -p rapidgzip-bench --bench inflate_backend --locked -- --quick --save-baseline zlib-rs
+      - run: cargo bench -p rapidgzip-bench --bench inflate_backend --features rapidgzip-core/isal --locked -- --quick --baseline zlib-rs
+
   msrv:
     runs-on: ubuntu-latest
     steps:

ARCHITECTURE.md

@@ -22,6 +22,106 @@ updates member accounting and calls the user's `Write`, so a writer need not be
 `Send`. `DecoderReader` substitutes a bounded synchronous channel at this final
 edge and therefore implements `Read + Send` without changing the decoder core.
 
+## Non-seekable input
+
+`rapidgzip-core` also accepts a plain `std::io::Read`. Such a source cannot be
+indexed, probed, or revisited, so paths 2 through 5 are all unreachable: each
+one begins by reading headers or block boundaries scattered across the file
+before it decodes anything. Path 1 is reachable, because it only ever moves
+forward.
+
+Both cursors implement one internal `InputCursor` trait, which is the exact set
+of forward operations path 1 and the member-header parser use: current offset,
+end of input, the readable window, consume, and confirm the source did not
+change. `SourceCursor` implements it over `ReadAt`; `StreamCursor` implements it
+over `Read` with a single window that compacts consumed bytes on refill. Member
+framing, footer verification, trailing-garbage detection, per-member history
+reset, and the output limit are therefore not reimplemented for streams: path 1
+is generic over the cursor and is the same code either way.
+
+The length snapshot does not exist for a non-seekable source rather than
+becoming mutable. A positional decode snapshots `len()` when it creates its
+cursor and re-checks it at the end, which is what `verify_source_unchanged`
+does. A stream has nothing to snapshot: end of input is whatever the reader
+reports, and the framing loop refuses to stop anywhere except at a verified
+member boundary, so trailing bytes after the last member are parsed as another
+header and rejected as trailing garbage. Reaching end of input therefore still
+means the whole input was verified, and `verify_source_unchanged` is a no-op.
+The `ReadAt` contract itself is unchanged, and no parallel path observes any of
+this.
+
+Sequencing follows the positional entry points. The first member header is
+validated against the initial window, without consuming it, before any
+coordinator is spawned. The runtime is then configured with a single worker so
+`DecoderStats` and `DecodeReport` report the concurrency actually in use.
+
+## Line counting
+
+`DecoderBuilder::count_lines` counts newlines in the `Output` implementations,
+on the thread that emits. That is the only place the bytes are final: the
+marker path's chunks hold 16-bit symbols until the coordinator resolves them,
+and a marker can resolve to a newline. Counting in the workers would therefore
+be wrong on the path that matters most.
+
+Checkpoint line offsets come from merging two ordered streams. The index
+builder keeps the offsets it has been offered but not yet passed, and each run
+of output resolves the ones it covers in a single scan. Offers always precede
+the emit of the bytes at their offset, so nothing is passed before it is known.
+The builder tracks whether every checkpoint was resolved and claims a total
+line count only when they all were, so a future path that offers late degrades
+to an index without counters rather than one full of zeros.
+
+## Structural analysis
+
+`Decoder::analyze` reuses `parallel/deflate.rs`, which already parses dynamic
+headers and decodes symbols. It needed the code lengths that parsing discards,
+so `dynamic_trees_with_lengths` hands them back. The decode loop is the
+analyzer's own: it counts symbol kinds and back-reference distances rather
+than producing markers, and it keeps plain output because it has no unknown
+history to represent.
+
+Header fields are parsed by the analyzer rather than by `parse_member_header`,
+which deliberately keeps only the offsets the decoder needs.
+
+The command-line report is a faithful reproduction of rapidgzip 0.16.0's, down
+to C++ number formatting and histogram bucketing, and a differential test
+diffs against the real tool. One derived field is deliberately not reproduced:
+rapidgzip's merged back-reference count comes from a pairwise merge after an
+unstable sort, which can shorten the current run and therefore depends on how
+equal distances were ordered. The analyzer computes a plain interval union
+instead.
+
+## Inflate backends
+
+Raw inflate sits behind one crate-internal trait, `InflateBackend`, with three
+operations: create, reset, and inflate a buffer into the spare capacity of an
+output vector. Its result is an enum, `Progress`, `StreamEnd`, or `Blocked`,
+rather than a zlib status code, so an implementation that does not speak zlib's
+numbering does not have to fake one. A type alias, `ActiveInflater`, picks the
+implementation at compile time: zlib-rs by default, ISA-L under the `isal`
+feature. Because the alias is a concrete type, the default build compiles to
+what it compiled to before the trait existed.
+
+Only the paths that decode a whole stream from its start go through the alias:
+sequential gzip members, the single-stream zlib and raw DEFLATE loop, and BGZF
+blocks. The marker/window path and `IndexedReader` name `RawInflater`
+directly. Both resume at arbitrary bit offsets, which needs `inflatePrime`, and
+the marker path locates DEFLATE block boundaries through zlib's `Z_BLOCK`
+contract. ISA-L exposes neither, so it must not be reachable from them, and the
+split is enforced by the type rather than by convention.
+
+The trait carries no dictionary call for the same reason. Every stream the
+pluggable paths inflate starts at its own beginning, with no predecessor
+history; installing a window belongs to the paths that resume mid-stream, and
+those keep the concrete type.
+
+Two properties of ISA-L shape its implementation. `inflate_state` embeds a
+64 KiB scratch buffer, so it is boxed rather than held inline. More
+importantly, ISA-L reads ahead into a bit buffer, so at the end of a stream it
+has consumed input the stream does not own; whole bytes still in that buffer
+are given back, or the gzip footer and the next member would be read from the
+wrong offset.
+
 ## Marker/window algorithm
 
 The implementation follows rapidgzip 0.16.0 at upstream commit
@@ -115,6 +215,65 @@ BGZF workers decode eight independently framed blocks per task directly into
 one aggregate output allocation, verify every block's CRC32 and ISIZE, and
 reuse their initialized zlib-rs stream with `inflateReset`.
 
+## Containers
+
+Framing is resolved before any path is selected. An explicit format is taken as
+given, since the framing checks report a mismatch far better than a prefix
+sniff; `Format::Auto` reads two bytes and falls back to reporting missing gzip
+magic, which is what it did before zlib was supported.
+
+gzip keeps every existing path. zlib and raw DEFLATE are each exactly one
+DEFLATE stream, so they skip the BGZF and multi-member probes and go to either
+`single_stream.rs`, which runs the sequential loop over an `InputCursor` and
+therefore serves positional and non-seekable sources identically, or the
+estimated-grid path when a worker budget exists.
+
+Supporting the parallel path costs three parameters rather than a second
+implementation: where DEFLATE starts, which checksum accumulates over the
+output, and what the end of the stream verifies. zlib validates its `CMF`/`FLG`
+header, accumulates Adler-32 through `libz-rs-sys`, and checks the trailer
+where a gzip footer would be read; raw DEFLATE starts at bit zero, accumulates
+nothing, and only refuses trailing bytes and, when the caller supplied one, a
+size that disagrees.
+
+Index checkpoints for these containers record the DEFLATE start rather than a
+header start, which is what indexed_gzip records, so `IndexedReader` resumes
+there directly.
+
+## Random-access index
+
+`index/` holds the index data model and the on-disk formats and knows nothing
+about decoding, so every format is testable against synthetic indexes. A
+checkpoint pairs a compressed bit offset with a decompressed byte offset and,
+unless it sits where no history is needed, the 32 KiB predecessor window that
+must become the inflate dictionary before resuming there. Windows are held
+zlib-compressed in memory by default, which matters once a large file
+accumulates thousands of them.
+
+Decode paths do not build the index themselves. They offer checkpoints to
+`RuntimeState`, which is already shared with every worker, and `IndexBuilder`
+orders, deduplicates, and thins the offers when the decode finishes. Offers may
+therefore arrive in any order, which is what lets concurrent workers contribute
+without coordination.
+
+What each path can offer differs. The estimated-grid path offers every chunk
+start together with the resolved predecessor window, so its checkpoints are
+interior points, usually not byte aligned. The BGZF path offers every non-empty
+block start with no window, reading each block's ISIZE footer for the
+decompressed offset, which is why a BGZF index exports as a complete htslib
+`.gzi`. The sequential and streaming paths offer member starts only: zlib does
+not report DEFLATE block boundaries, so a forward-only source yields a coarse
+but valid index.
+
+`indexed/` consumes an index. `IndexedReader` picks the last checkpoint at or
+before the target, primes the inflater with the straddled bits when the
+checkpoint is not byte aligned, installs the window, and discards output up to
+the target. A checkpoint with no window is not assumed to be a member start:
+indexed_gzip records its first point after the gzip header, so the reader
+checks for the magic bytes and skips a header only when one is there. Expanded
+windows are cached in a byte-bounded LRU so nearby seeks do not re-inflate the
+same history.
+
 ## Scheduling and memory
 
 The BGZF, stored, dense-member, and native paths use a
@@ -188,3 +347,12 @@ oversized regions continue through zlib-rs instead. `DecoderReader` adds at most
 the configured in-flight chunk count plus its currently partially consumed
 chunk. Dropping it closes the consumer edge, sets cancellation, and joins the
 coordinator.
+
+A non-seekable source holds one input window instead of a positional page and
+spools nothing, so its memory is independent of the input length. Backpressure
+reaches the producer without any new mechanism: the bounded final handoff blocks
+the coordinator, the coordinator therefore stops reading, and the pipe fills.
+Dropping such a reader closes the consumer edge and sets cancellation as usual,
+but does not join. Its coordinator can be parked inside a read against a
+producer that never writes again, and a drop that could block forever is worse
+than a thread that exits at its next read or send boundary.

CHANGELOG.md

@@ -3,6 +3,83 @@
 All notable changes to this project are documented in this file. The project
 uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- Two gzip index interoperability bugs, found by a test in which real gztool
+  extracts by line from an index this crate wrote. gztool numbers lines from
+  one while the index counts newlines before a point, so export and import now
+  convert. More seriously, a member checkpoint recorded the member start, so
+  gztool resumed by inflating raw DEFLATE at the gzip header and reported a
+  compressed data error. Member checkpoints now record the DEFLATE start,
+  matching indexed_gzip and gztool. BGZF block checkpoints still record the
+  block start, which is what `.gzi` means by an offset.
+- Writing gztool's line-aware format from an index that never counted lines
+  wrote zero for every counter, producing a file gztool accepts and then
+  trusts. It is now refused.
+
+### Added
+
+- Structural analysis. `Decoder::analyze` walks every DEFLATE block and
+  returns an `Analysis`: per stream the container, header fields, offsets and
+  footer; per block the encoding, offsets to the bit, sizes, declared Huffman
+  alphabets, symbol-kind counts, and how far its back-references reach into
+  the preceding window. The CLI prints it through `--analyze` in rapidgzip
+  0.16.0's exact layout, checked by diffing against the real tool in CI.
+- Newline counting, through `DecoderBuilder::count_lines`. It fills
+  `DecodeReport::line_count` and, when an index is also collected, a line
+  offset for every checkpoint plus the index total.
+  `IndexedReader::seek_to_line` seeks by zero-based line number, refusing an
+  index that carries no counters rather than scanning from the start.
+- Command-line parity with rapidgzip 0.16.0. `rapidgzip-rust` now accepts
+  every option that tool accepts, under the same names: `--ranges` with byte
+  and line addressing, `--import-index` and `--export-index` with
+  `--index-format`, `--count`, `--count-lines`, `--format`, `-f`, `-k`, `-d`,
+  `--chunk-size`, `-q`, `-v`, and `--oss-attributions`. Output goes where
+  rapidgzip sends it, including the derived name with the compressed suffix
+  stripped. `--io-read-method` and `--sparse-windows` are accepted no-ops;
+  `--no-verify` is refused, since verification here is structural.
+- An optional ISA-L raw-inflate backend, behind the off-by-default `isal`
+  feature of `rapidgzip-core`. It replaces zlib-rs on the paths that decode a
+  whole stream from its start: sequential gzip members, single-stream zlib and
+  raw DEFLATE, and BGZF blocks. The parallel marker/window path and
+  `IndexedReader` stay on zlib-rs either way, since both resume at arbitrary
+  bit offsets and the parallel path needs zlib's `Z_BLOCK` contract. The
+  feature links a system `libisal` rather than building one. Default builds are
+  unchanged in dependencies and behaviour. Whether ISA-L is faster is a
+  measurement; `README.md` reports it.
+- zlib (RFC 1950) and raw DEFLATE (RFC 1951) decoding, selected through
+  `DecoderBuilder::format`. `Format::Auto`, the default, detects gzip against
+  zlib; raw DEFLATE must be requested. Both containers decode sequentially, in
+  parallel, from non-seekable input, and with a random-access index.
+  `DecoderBuilder::expected_uncompressed_size` verifies raw DEFLATE output,
+  which carries no checksum of its own. `DecodeReport` gains `format`.
+- Random-access indexing. `DecoderBuilder::build_index` collects a `GzipIndex`
+  during an ordinary decode and returns it in `DecodeReport::index`.
+  `DecoderBuilder::index_spacing` and `DecoderBuilder::compress_index_windows`
+  tune checkpoint density and resident memory.
+- `IndexedReader`, a `Read + Seek` view over compressed input that resumes at
+  the nearest checkpoint instead of decoding from the start.
+- Index persistence in four formats, importing and exporting: the crate's own
+  versioned format, indexed_gzip `GZIDX`, htslib BGZF `.gzi`, and gztool.
+  Interoperability with all three tools is covered by tests.
+
+- Decoding of non-seekable compressed input through `Decoder::decode_stream`
+  and `Decoder::stream_reader`, which accept any `std::io::Read` and mirror
+  `Decoder::decode` and `Decoder::reader`. `Decoder::open` now routes a path
+  that cannot be read positionally, such as a FIFO, character device, or
+  socket, to the streaming decoder instead of failing, and the CLI accepts `-`
+  for standard input. Such input runs the sequential zlib-rs path that the
+  parallel paths already use as their authoritative fallback, sharing its
+  framing, footer verification, trailing-garbage detection, and output limit,
+  so it is verified identically but is not decoded in parallel.
+  `DecoderStats` and `DecodeReport` report a single worker for it rather than
+  the configured thread budget. Nothing is spooled; input memory is one
+  configured input window. Dropping a streaming `DecoderReader` before end of
+  output cancels without waiting for its background thread, so a stalled
+  producer cannot block the drop.
+
 ## [0.1.0] - 2026-07-31
 
 Initial release of the decoder-only `rapidgzip-rust` implementation.
@@ -54,4 +131,5 @@ Initial release of the decoder-only `rapidgzip-rust` implementation.
   cell; multi-worker parity and the zlib-ng-backed C++ performance gate are
   met on the published workloads.
 
+[Unreleased]: https://github.com/COMBINE-lab/rapidgzip-rust/compare/v0.1.0...HEAD
 [0.1.0]: https://github.com/COMBINE-lab/rapidgzip-rust/releases/tag/v0.1.0