DEFLATE analysis and the --analyze report

[BenjaminDEMAILLE]

Sub-project 4b of 4, the last one, replacing #5. Stacked on #11, which stacks on #10, #9, #8. GitHub cannot base a cross-fork pull request on a fork branch, so this targets main and its diff contains all four parents; review the top four commits only. Merge #8, #9, #10, and #11 first.

What this adds

Decoder::analyze walks every DEFLATE block and returns an Analysis: per stream the container, its header fields, offsets and footer; per block the encoding, offsets to the bit, sizes, the three declared Huffman alphabets, how much output came from literals against copies, and how far its back-references reach into the preceding window.

--analyze prints it in rapidgzip 0.16.0's exact layout.

The walk reuses parallel/deflate.rs, which already parses dynamic headers and decodes symbols. It needed the code lengths that parsing discards, so dynamic_trees_with_lengths hands them back. The decode loop is the analyzer's own, since it counts symbol kinds and back-reference distances rather than producing markers.

How byte-exact it actually is

Diffed against the real tool on a 754 KiB gzip file: 974 lines of report, and only two things differ.

The benchmark profile prints wall-clock durations. Ours carries its own measurements, so the section is there with the same labels and shape but different numbers.

Number of merged back-references differs in some blocks, and this one is worth explaining because it is not a bug on our side. The reference sorts references by distance with std::sort, then merges pairwise, assigning the current run the following reference's end even when that reference is contained, which shortens the run. Its result therefore depends on how its unstable sort ordered equal distances, so it is not a function of the file. Sorting by distance then length ascending and descending both fail to match it, in opposite directions, which is what confirmed the cause. Ours is a plain interval union: deterministic, and where they disagree, correct.

Everything else matches byte for byte, including the parts most likely to be subtly wrong: 31 KiB 180 B style byte formatting, %.6g and %.6e number formatting, and the histogram bucketing with its two easy-to-miss rules (an integral range narrower than the bin count shrinks the bin count, and the maximum lands in the last bin).

Two sections the reference contains are deliberately not printed. Its back-reference-length and window-symbol histograms are guarded on a counter its histogram type never updates, so neither can ever appear in real output. Emitting them would be a difference, not an improvement.

Verification

• tests/analyze_interop.rs, ignored by default, diffs against real rapidgzip 0.16.0 on a single member, a stored member, concatenated members, and a six-byte file. The interop CI job installs it from PyPI.
• 9 core tests asserting what a wrong walk would violate: block sizes summing to the decoded total, offsets that only increase, literals plus copies equalling block output, a final block ending every stream, across gzip, multi-member, BGZF, stored, zlib, and raw DEFLATE.
• 19 unit tests on the C++ formatting reproductions, which is where a rounding rule that is wrong only at an edge case would otherwise surface as an opaque diff.
• fmt, clippy, rustdoc, and the full suite green.

This completes the replacement of #5

Four sub-projects, five pull requests: #8 index and seeking, #9 zlib and raw DEFLATE, #10 pluggable inflate backends, #11 command-line parity, and this one.

[rob-p]

Status update after the current-main review and #18

The analyzer is a worthwhile feature, especially as a diagnostic for unusual DEFLATE structure, but this stacked PR should not be merged into current main. Its core feature can be implemented independently of the CLI-parity work in #11 and independently of #18; the final --analyze switch can follow after the structured core API lands.

The clean design should retain the strong parts of this proposal—one authoritative block walk, structured library output, multi-format/multi-member coverage, and differential checks against C++ rapidgzip—while changing the following points.

Recommended API and resource model

• Keep Analysis as structured facts, not presentation. C++ number formatting, histogram layout, section suppression, and compatibility quirks belong in the CLI crate.
• Keep measured wall-clock durations out of deterministic structural equality. Return timings as a separate optional AnalysisTimings value, or let the CLI time phases while rendering the benchmark-profile section.
• Replace unbounded retain_backreferences: bool with an explicit retention budget. Summaries are always collected; detailed references are retained only up to a caller-supplied maximum, with the result recording retained and omitted counts. A hostile stream must not turn --analyze --verbose into unbounded memory growth.
• Use checked counters and bounded try_reserve allocation for block, alphabet, and optional-reference collections. Return typed analysis errors through the existing error model rather than panicking or saturating silently.
• Share framing validation and format selection with the current gzip/zlib/raw implementation. If analysis needs header fields the fast parser currently discards, add a detailed parser/view beside it rather than maintaining a second parser with different FHCRC, reserved-flag, truncation, or BGZF rules.
• Keep the block walker single-threaded. Its dictionary and decoded offsets are causally ordered; parallelizing the reporting layer would add complexity without independent block histories.

Reference-output compatibility

The CLI should target rapidgzip 0.16.0's report where that report is a deterministic function of the input, with fixtures for exact bytes. Known reference defects should be normalized and documented rather than copied into the public data model. In particular, deterministic interval union is preferable to reproducing unstable equal-distance sort behavior for merged back-references.

The compatibility test should mask or compare separately:

• locally measured benchmark timings;
• the documented merged-reference defect; and
• sections that the reference implementation cannot emit because of its own dead guard.

That makes the claim precise: deterministic sections match, measured sections have matching shape, and deliberate corrections are named.

Clean implementation sequence

1. Add internal reusable block-walking primitives on current main, preserving the existing hot marker decoder and its SIMD behavior.
2. Add documented non-exhaustive structured analysis types plus bounded AnalyzeOptions, with no CLI formatting dependency.
3. Validate gzip optional headers, concatenated/empty members, BGZF, stored/fixed/dynamic blocks, zlib, raw DEFLATE, truncation, trailers, and trailing data. Assert monotonic bit offsets and exact encoded/decoded totals.
4. Add CLI-only C++ formatting and histogram helpers with edge-case unit tests.
5. Add --analyze, compatibility fixtures, and an optional ignored differential test against pinned rapidgzip 0.16.0.
6. Document complexity and retention bounds, then run fmt, Clippy with warnings denied, full tests, doctests/rustdoc, and interop CI.

#18 does not need analyzer code. In the future, the same back-reference summaries may inform a real sparse-window index transformation, but that should be a separate measured feature rather than coupling analysis to indexed full-stream decoding now.

I suggest keeping this PR as the reference-output research artifact until a clean core-analysis PR is opened, then closing the stack as superseded.

[rob-p]

Superseded by #20, which reimplements the structural-analysis feature cleanly from the current main branch.

The replacement keeps the useful analysis and compatible --analyze report while adding bounded result retention, positional and streaming library APIs, shared authoritative framing, deterministic interval unions, full multi-member/BGZF handling, and explicit safety documentation. It also reaches parity on the retained FASTQ workload: 1.70 s for Rust versus 1.71 s for rapidgzip 0.16.0 with ISA-L in the five-run comparison.

Closing this stacked version so review and CI can focus on the current implementation. Thank you for the original design and compatibility work; it directly informed the replacement.