Command-line parity with rapidgzip, and line counting

[BenjaminDEMAILLE]

Sub-project 4a of 4 replacing #5. Stacked on #10, which stacks on #9, which stacks on #8. GitHub cannot base a cross-fork pull request on a fork branch, so this targets main and its diff contains all three parents; review the top four commits only. Merge #8, #9, and #10 first.

--analyze is deliberately not here. It needs a DEFLATE block walker in the core library plus a byte-exact reimplementation of rapidgzip's report, which together are the same size again. That is sub-project 4b, already designed in docs/superpowers/specs/2026-08-01-analyze-design.md.

The option surface

Every option rapidgzip 0.16.0 accepts is now accepted here under the same short and long names, so a command line written for that tool works. The work already in the library becomes reachable without writing Rust: indexes in five formats, seeking, zlib and raw DEFLATE.

main.rs kept only arguments and dispatch. Input classification and output destination went to source.rs, index formats to index.rs, range parsing and extraction to ranges.rs, result printing to report.rs. The two pieces with real logic, range parsing and output path derivation, are pure functions with unit tests beside them.

Three options are accepted no-ops because they name behaviour this crate does not have: --io-read-method, which has one strategy here, and --sparse-windows with its negation, since index windows are always dense. A dense index is valid and interoperable, merely larger, so accepting them lies about nothing.

--no-verify is refused instead. Verification is structural here: a member is accepted only after its CRC32 and size check out. Accepting the flag would promise a speedup that does not exist.

Two more refusals replace silent surprises: --import-index without --ranges, where the index would go unused, and --index-format gztool-with-lines without --count-lines, where the format has nowhere to get its counters.

Line counting, and the bug that made it necessary

Checkpoint::line_offset and GzipIndex::total_line_count already existed and were read by the gztool importer. Every decode path wrote zero. A gztool-with-lines index exported from here was silently wrong.

DecoderBuilder::count_lines fills them. Counting happens in the Output implementations, on the thread that emits, because that is the only place the bytes are final: the marker path's chunks hold 16-bit symbols until the coordinator resolves them, and a marker can resolve to a newline.

Checkpoint line offsets come from merging two ordered streams. The builder keeps the offsets it has been offered but not yet passed; each run of output resolves the ones it covers in one scan. It tracks whether every checkpoint was resolved and claims a total only when they all were, so a future path that offers late degrades to an index without counters rather than one full of zeros.

IndexedReader::seek_to_line seeks by zero-based line, reading at most one checkpoint spacing regardless of file size, and refuses an index without counters rather than scanning from the start.

Two more interop bugs, found by a test

Adding a test where real gztool extracts by line from an index we wrote found two bugs the existing byte-offset test could not see.

gztool numbers lines from one; our offsets count newlines before a point. Export and import now convert.

More seriously, a member checkpoint recorded the member start, so gztool resumed by inflating raw DEFLATE at the gzip header and reported Compressed data error. indexed_gzip and gztool both record where DEFLATE begins. Member checkpoints now do too. BGZF block checkpoints still record the block start, which is what .gzi means by an offset.

The old test missed this because it only ever seeks to an interior checkpoint, which carries a window and a bit offset and was always correct. Only the empty-window member points were wrong.

Verification

• 18 CLI integration tests driving the built binary through CARGO_BIN_EXE_rapidgzip-rust, no new dependency.
• 9 unit tests for range parsing and output path derivation.
• 11 line-counting tests asserting the property the format actually promises: each checkpoint's line offset equals the newlines preceding its decompressed offset, on both the marker grid and BGZF.
• All 7 index interop tests pass against real bgzip, indexed_gzip, and gztool 1.8.2, in both directions, including the new extract-by-line case.
• fmt, clippy, rustdoc, and the full suite green.

[rob-p]

Status update after the current-main review and #18

The feature set remains worthwhile, but this stacked branch should not be merged into the current head. #8 and #9 landed through redesigned implementations, the optional-backend stack was not adopted, and several old index/model fixes in this diff are now either obsolete or conflict with the format-neutral current API.

#18 also changes one important CLI decision: --import-index without --ranges is no longer an unused-index error. Once #18 lands, the CLI should use the imported index for strict full-stream parallel decoding.

The clean implementation should be split into two reviewable changes.

1. Line metadata and line seeking in the core

• Keep counting opt-in and perform it only over final ordered bytes, after marker resolution.
• Preserve DecodeReport: Copy; an optional scalar line count is compatible with that guarantee and does not put an owning index in the ordinary report.
• Merge ordered output offsets with authoritative checkpoints from the current DeflateIndex/CheckpointKind model. Do not carry forward the old member-header/payload-offset fixes, because current main already represents both explicitly.
• Set DeflateIndex::total_line_count and per-checkpoint line_offset only when every value is known. gztool-with-lines export must refuse missing metadata rather than synthesize zeroes.
• Add line seeking to IndexedReader with a clear zero-based public convention and explicit conversion only at the gztool codec boundary.
• Cover gzip, concatenated and empty members, BGZF, zlib/raw counting, sequential/marker/streaming paths, no-final-newline input, backward/repeated line seeks, and real gztool interoperability.

This can land independently before the CLI surface.

2. CLI parity on current main

• Retain the proposed module split: source/output routing, index formats, range parsing, reporting, and attributions should not accumulate in main.rs.
• Implement flags against current Format, DeflateIndex, IndexedDecodeReport, IndexingDecoderReader, and typed error APIs rather than adapting the old stacked types.
• Treat compatibility flags in three documented categories: real behavior, harmless syntax aliases, and unsupported semantic requests. Continue rejecting --no-verify. For options such as sparse-window transformation that name behavior we do not implement, an explicit unsupported error is safer than a silent promise unless we can demonstrate that reference scripts depend on the no-op.
• Keep -d and -k as harmless compatibility aliases because this binary is decode-only and never removes input.
• Preserve automatic -P 0 as a computed worker budget, then let the runtime controller and application ceiling decide actual allocation.

The index routing should now be:

• imported index plus byte/line ranges: IndexedReader;
• imported index plus ordinary full output/test/count: Decode full streams in parallel from an existing index #18's decode_from_index or reader_from_index;
• export without import: an explicit indexing decode followed by the requested codec;
• line ranges or gztool-with-lines: require real line metadata and explain its absence;
• non-seekable input: reject operations requiring positional reuse, while retaining coarse forward index construction only where the current API supports it.

Full-stream imported-index failure must remain strict. A malformed, source-size-mismatched, wrong-format, or exact-boundary-mismatched index should produce IndexDecodeError and a nonzero exit, never fall back and appear successful.

Validation plan

1. Land the focused core line-count/line-seek PR with unit, property, format, and gztool interop coverage.
2. Rebuild the CLI modules from current main in a separate PR.
3. Test every output destination and overwrite rule, stdin restrictions, supported format, index codec, import/export combination, byte and line range form, count mode, error exit, broken pipe, and quiet/verbose behavior through the built executable.
4. Add a full-output imported-index test that asserts DecoderPath::IndexedParallel, byte identity, and strict rejection of an index for another same-sized archive.
5. Run fmt, Clippy with warnings denied, full tests, doctests/rustdoc, and the available indexed_gzip/gztool/bgzip interoperability jobs.

I suggest retaining this PR as the design and compatibility reference until the two clean successors are opened, then closing it as superseded rather than attempting to rebase or cherry-pick the stack.

[rob-p]

Superseded by #19, a clean implementation based on current main after #18. It retains the line-aware indexing and CLI goals while using DeflateIndex, strict imported-index decoding, all current format paths, explicit rejection of unsupported modes, vectorized ordered counting, and the current scheduler. Thank you for the original design and interoperability work; it directly informed the replacement.