Merge pull request #19 from COMBINE-lab/line-aware-cli

Author: Rob Patro

ARCHITECTURE.md

@@ -119,6 +119,51 @@ across interior checkpoints, validates every trailer, and reports
 `DecoderPath::IndexedParallel`. The complete design is recorded in
 `docs/design/indexed-parallel-decode.md`.
 
+## Ordered line counting and line seeking
+
+Newline counting is decoder configuration, while index construction remains an
+explicit per-operation choice. `DecoderBuilder::count_lines(true)` adds one
+scalar scan over final ordered output and places the result in
+`DecodeReport::line_count`; the field is `Option<u64>`, so `DecodeReport`
+remains `Copy`. The disabled path keeps no counter and does not scan output.
+
+Counting occurs at the `Output` boundary after predecessor markers have been
+resolved. Worker-local or speculative buffers are not valid inputs because an
+unresolved symbol may later become a newline. Push decodes wrap their final
+output sink in a decode-local counter. Pull-driven streaming uses the same
+counter immediately before publishing each final chunk. Strict indexed
+parallel decoding counts its ordered span handoff.
+
+When indexing and counting are both requested, `IndexCollector` advances one
+cursor through the retained checkpoint vector as ordered byte ranges arrive.
+It annotates checkpoints in place, avoiding duplicate tree nodes and logarithmic
+lookup work per point. Checkpoints at decoded EOF are resolved during
+finalization. Empty gzip members may produce multiple checkpoints at the same
+output offset; they correctly share one line count. If any retained checkpoint
+was offered too late to resolve, finalization clears every checkpoint line
+field and the total rather than publishing partial metadata.
+
+`IndexedReader::seek_to_line` selects the latest complete annotated checkpoint
+proven to precede the requested zero-based line's start, resumes through its
+exact predecessor window, and scans forward to the terminating newline. It
+uses a strictly smaller line count for nonzero targets because an equal-count
+checkpoint can sit inside a long line. The scan is bounded by the actual
+decoded distance to the previous retained checkpoint for an in-range target;
+external, streaming-built, or thinned indexes may have gaps larger than their
+recorded target spacing. A request beyond the final line lands at
+decoded EOF. The method rejects indexes without a total and per-point line
+counters. gztool version 1 is translated at the codec boundary because it
+stores one-based checkpoint line numbers, while the Rust API stores preceding
+newline counts.
+
+The complete design and CLI integration are recorded in
+`docs/design/line-aware-cli.md`.
+
+For strict full-stream decoding through an imported index, enabling line
+counting also authenticates every supplied checkpoint line offset and the
+total against final ordered bytes. Random-access seeking alone must trust those
+annotations because it intentionally does not scan the skipped prefix.
+
 ## Marker/window algorithm
 
 Before the generic marker grid starts, path admission derives a useful worker

BENCHMARKING.md

@@ -15,6 +15,43 @@ ratio between the two groups isolates indexing overhead on identical data.
 Record checkpoint count, serialized native-index size, and peak RSS alongside
 throughput when evaluating a release candidate.
 
+The paired `bgzf_line_count` and `bgzf_line_count_with_index` groups use a
+32 MiB FASTQ-like stream split into 4 KiB BGZF blocks (more than eight thousand
+retained checkpoints). Their ratio isolates checkpoint annotation from SIMD
+newline counting, while peak RSS from the benchmark process exposes growth per
+checkpoint. The index builder annotates its checkpoint vector in place and
+must not recreate the former tree-node allocation per checkpoint.
+
+For a larger peak-RSS diagnostic without Criterion's sampling allocations:
+
+```bash
+cargo build --release -p rapidgzip-bench --bin line_index
+/usr/bin/time -v target/release/line_index count 8 128
+/usr/bin/time -v target/release/line_index index 8 128
+```
+
+Both modes generate the same 128 MiB stored-BGZF fixture with 4 KiB blocks and
+discard generation scratch before timing. The index mode retains more than
+32,000 line-annotated checkpoints. Compare elapsed time and maximum resident
+set size across repeated, alternating invocations.
+
+### 2026-08-03 line-index diagnostic
+
+On the dual-socket Xeon E5-2699 v4 host described below, five alternating
+un-pinned runs of the command above at eight workers produced these medians:
+
+| mode | decoded bytes | checkpoints | timed decode | peak RSS |
+|---|---:|---:|---:|---:|
+| line count | 134,217,728 | 0 | 48.362 ms | 136,700 KiB |
+| line count + index | 134,217,728 | 32,768 | 51.518 ms | 138,088 KiB |
+
+In-place annotation therefore added 6.5% elapsed time and 1,388 KiB peak RSS,
+or about 43 bytes per retained checkpoint, on this deliberately dense index.
+The larger quick Criterion sweep measured count-plus-index overhead of 11.9%,
+11.9%, and 5.9% at 1, 4, and 16 workers respectively; those single-sample
+figures are retained as a routing diagnostic rather than a portable speed
+claim.
+
 The `decoder_reader_deflate_formats` group compresses one identical payload as
 gzip, zlib, and raw DEFLATE, then decodes each through the public reader at the
 same worker budgets. It is a paired container-overhead regression check, not a
@@ -450,3 +487,23 @@ FASTQ contains many small internal DEFLATE blocks, so multi-worker throughput
 fell below 350 MiB/s despite synthetic success. Accumulating internal block
 results into configured output chunks restored scaling; a focused integration
 test now enforces a chunk-plus-span bound on writer handoffs.
+
+## 2026-08-03 line-counting FASTQ diagnostic
+
+The line-aware CLI successor was checked on the same public FASTQ file and
+dual-socket host described above. This was an unpinned implementation
+diagnostic, not a release parity matrix. The release CLI used a 16-worker
+budget, one warmup per mode, and five alternating `/usr/bin/time` runs. Both
+modes decoded and verified the full stream to a sink; the counted mode reported
+4,320,920 newline bytes.
+
+| mode | five wall times (s) | median (s) | decoded MiB/s |
+|---|---|---:|---:|
+| counting disabled (`--test`) | 0.25, 0.27, 0.25, 0.25, 0.24 | 0.25 | 1,380.2 |
+| counting enabled (`--count-lines`) | 0.28, 0.28, 0.28, 0.27, 0.27 | 0.28 | 1,232.3 |
+
+The first scalar implementation measured 0.52 seconds in the counted mode on
+the same warm cache, more than twice the 0.25-second control. Runtime-dispatched
+AVX2 with SSE2 and NEON baselines reduced the measured optional cost to roughly
+12%. The ordinary disabled path still performs no scan and only tests the
+decode-local enabled flag at each ordered output chunk.

CHANGELOG.md

@@ -5,8 +5,40 @@ uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Fixed
+
+- Parallel task queues now publish their availability counters before making
+  work visible, preventing a worker from consuming a task before the matching
+  counter increment.
+- CLI index import now streams through the core allocation limits, requires an
+  exact format length, and rejects trailing data instead of buffering an
+  untrusted file or treating arbitrary bytes as an empty GZI.
+- Index export now serializes transactionally through a same-directory
+  temporary file, preserving an existing destination after incompatibility or
+  write failures. Normal decode output treats an early closed pipe as a
+  successful consumer exit through wrapped decoder errors as well as direct
+  I/O errors.
+- Output collision checks now compare the already-open input file with the
+  destination by file identity on Unix and Windows, preventing hard-link and
+  symlink aliases from truncating a forced input.
+
 ### Changed
 
+- The `rapidgzip-rust` CLI now exposes rapidgzip-compatible decoding, counting,
+  index import/export, range extraction, output, format, and reporting options.
+  Imported indexes drive strict full-stream indexed decoding. Options whose
+  semantics are not implemented, including disabled verification, sparse
+  windows, and shared-cursor I/O strategies, are rejected instead of ignored.
+- The CLI's format-neutral index default is now native. Payload output can be
+  combined with byte and line counting in one pass; reports move to stderr when
+  stdout carries decoded bytes. Imported range extraction documents its
+  partial-verification boundary, while `--verify` performs a complete strict
+  pass and otherwise-ignored decoder options are rejected.
+- Line-aware index construction annotates the retained checkpoint vector in
+  place instead of duplicating every point in ordered tree structures. Strict
+  indexed decoding with line counting enabled now authenticates imported
+  checkpoint and total line counters.
+
 - Generic gzip, zlib, and raw-DEFLATE path selection now uses a bounded
   machine-, runtime-budget-, task-count-, and input-aware admission screen.
   It replaces a fixed low-thread marker/window policy, keeps short inputs and
@@ -20,6 +52,21 @@ uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Added
 
+- Opt-in newline counting through `DecoderBuilder::count_lines` and the new
+  `DecodeReport::line_count` scalar, preserving `DecodeReport: Copy`. Combining
+  counting with explicit index construction annotates every retained
+  checkpoint and the index total on final ordered output. Concatenated and
+  empty gzip members, BGZF, zlib, raw DEFLATE, streaming push/pull, marker
+  decoding, and strict indexed decoding share the same semantics.
+- `DeflateIndex::checkpoint_at_or_before_line` and
+  `IndexedReader::seek_to_line` for zero-based line access. gztool version 1
+  import/export translates its one-based checkpoint numbering and refuses
+  incomplete line metadata.
+- CLI byte and line ranges using rapidgzip's comma-separated `SIZE@OFFSET`
+  syntax, including binary byte units, `L`, `inf`, overlaps, and ordered
+  extraction. The CLI can read and write native, GZIDX, gztool, line-aware
+  gztool, and BGZF `.gzi` indexes.
+
 - Strict parallel full-stream reuse of caller-supplied `DeflateIndex` values
   through `Decoder::decode_from_index` and `Decoder::reader_from_index`. Every
   span must match its next compressed-bit and decompressed-byte checkpoint;

Cargo.lock

@@ -23,6 +23,7 @@ libz-rs-sys = "0.6.6"
 paraseq = { version = "0.4.14", default-features = false }
 proptest = "1.11.0"
 rapidgzip-core = { path = "crates/rapidgzip-core", version = "0.1.0" }
+same-file = "1.0.6"
 
 [workspace.lints.rust]
 unsafe_op_in_unsafe_fn = "deny"

Cargo.toml

@@ -16,6 +16,8 @@ The project provides:
   stream suitable for parsers such as [paraseq];
 - opt-in random-access index construction, interoperable index formats, and a
   decoded-output `Read + Seek` adapter;
+- opt-in newline counting, line-annotated indexes, and indexed seeking by
+  zero-based line number;
 - decoding of non-seekable compressed input such as standard input, a FIFO, a
   process substitution, or a socket.
 
@@ -321,7 +323,51 @@ Format parsers apply explicit checkpoint and window-allocation limits through
 `IndexReadOptions`. The native format represents every container. `.gzi`
 export requires an index proven to come from BGZF; GZIDX and gztool export
 require gzip-family provenance; gztool line-aware export requires real line
-counters and never invents them.
+counters and never invents them. The CLI detects formats from a bounded prefix,
+streams index parsing, rejects trailing bytes, and writes exports through a
+same-directory temporary file so failed conversions cannot truncate an
+existing index.
+
+Line metadata is collected only when requested. Enabling
+[`DecoderBuilder::count_lines`] adds `DecodeReport::line_count`; combining it
+with an explicit indexing operation also annotates every retained checkpoint:
+
+```rust,no_run
+use rapidgzip_core::{Decoder, IndexOptions, IndexedReader};
+use std::fs::File;
+use std::io;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let decoder = Decoder::builder()
+        .decoder_threads(8)
+        .count_lines(true)
+        .build()?;
+    let source = File::open("reads.fastq.gz")?;
+    let indexed = decoder.decode_with_index(
+        &source,
+        &mut io::sink(),
+        IndexOptions::default(),
+    )?;
+    assert_eq!(indexed.decode.line_count, indexed.index.total_line_count());
+
+    let mut reader = IndexedReader::new(source, indexed.index)?;
+    let byte_offset = reader.seek_to_line(1_000_000)?;
+    println!("line 1000000 begins at decoded byte {byte_offset}");
+    Ok(())
+}
+```
+
+A line offset is the number of `b'\n'` bytes preceding a position. Line zero
+begins at decoded byte zero. A final unterminated line does not increase
+`line_count`, while `seek_to_line` can still reach it after scanning from the
+nearest checkpoint. Counting happens once on final ordered bytes, after marker
+resolution, and is disabled by default. `DecodeReport` remains `Copy` because
+the result is an optional scalar. A line-aware index is published only when
+every retained checkpoint received an exact count; partial metadata is never
+presented as complete. Strict full-stream decoding with line counting enabled
+also recomputes imported per-checkpoint and total line counts, rejecting
+structurally valid but incorrect navigation metadata. Without line counting,
+imported line offsets remain explicitly trusted navigation data.
 
 `IndexedReader` validates the index and any recorded source size before use.
 Resuming at a gzip member or zlib-header checkpoint verifies that complete
@@ -346,6 +392,29 @@ rapidgzip-rust -P 16 reads.fastq.gz > reads.fastq
 # Verify every member and discard decoded output.
 rapidgzip-rust -P 16 --test reads.fastq.gz
 
+# Count decoded bytes or newline bytes.
+rapidgzip-rust --count reads.fastq.gz
+rapidgzip-rust --count-lines reads.fastq.gz
+
+# Decode and count in one pass. Counts use stderr when payload uses stdout.
+rapidgzip-rust -c --count --count-lines reads.fastq.gz > reads.fastq
+
+# Build a native index, then use it for strict full-stream parallel decoding.
+rapidgzip-rust --test --export-index reads.rgzidx \
+  --index-format native reads.fastq.gz
+rapidgzip-rust --import-index reads.rgzidx -c reads.fastq.gz > reads.fastq
+
+# Extract byte and zero-based line ranges in the requested order.
+rapidgzip-rust --ranges '1KiB@4MiB,10L@1000L' -c reads.fastq.gz
+
+# Authenticate the complete source before an imported random-access read.
+rapidgzip-rust --import-index reads.rgzidx --verify \
+  --ranges '10L@1000L' -c reads.fastq.gz
+
+# Export gztool version 1 with real line counters.
+rapidgzip-rust --count-lines --export-index reads.gzi \
+  --index-format gztool-with-lines reads.fastq.gz
+
 # Refuse to overwrite an existing output file.
 rapidgzip-rust -P 16 --output reads.fastq reads.fastq.gz
 
@@ -356,12 +425,36 @@ cat reads.fastq.gz | rapidgzip-rust - > reads.fastq
 rapidgzip-rust <(some_producer) > reads.fastq
 ```
 
-`-P`/`--threads` is a maximum decoder-worker budget. Parallel paths bootstrap
+The CLI auto-detects gzip or zlib by default; raw DEFLATE requires
+`--format raw-deflate`. `--chunk-size` controls the decoded handoff size in
+KiB. Output defaults to stdout when redirected; at a terminal, a regular input
+derives a safe output filename using case-insensitive compression suffixes.
+Existing files require `--force`. Index export defaults to the native,
+format-neutral representation; gzip-specific interoperable formats must be
+selected explicitly.
+
+`-P`/`--decoder-parallelism` (`--threads` is an alias) is a maximum
+decoder-worker budget. Parallel paths bootstrap
 from the smaller of the affinity-visible processors and this requested budget,
 then create more workers only while measurements justify them. They may retain
 fewer active workers when the input exposes less parallel work, the consumer is
 backpressured, or additional concurrency reduces throughput.
 
+Imported indexes are never advisory. Full-stream operations use
+`decode_from_index`, and malformed, incomplete, or source-mismatched indexes
+fail without falling back. Range extraction uses `IndexedReader`; a line range
+requires complete line metadata in an imported index or builds a line-aware
+index first. A seek from an interior DEFLATE checkpoint cannot authenticate
+the skipped prefix. For that reason, `--verify` on an imported range performs
+a complete strict indexed decode before extraction. Decoder options that would
+otherwise be ignored by an unverified imported range are rejected and explain
+that `--verify` is required. `--no-verify`, `--sparse-windows`, and the `sequential` and
+`locked-read` I/O methods are rejected because the current implementation
+cannot honor their semantics. Outside imported ranges, complete decode paths
+already verify their selected framing. `--no-sparse-windows`, `-d`, and `-k`
+remain compatibility aliases. This is a deliberately compatible subset, not a
+claim that every rapidgzip CLI option is implemented.
+
 ## Correctness and resource behavior
 
 - Every accepted gzip member is terminated by an actual final DEFLATE block
@@ -439,7 +532,8 @@ The integration suite covers gzip, zlib, raw DEFLATE, multi-member streams,
 BGZF, corruption, format detection across short/interrupted reads, false header
 candidates, output limits and exact sizes, index construction, indexed
 full-stream parallel decode, seeking, cancellation, one-byte consumer buffers,
-and direct paraseq consumption. Generated benchmark corpora and large
+line counting and seeking, CLI index/range workflows, and direct paraseq
+consumption. Generated benchmark corpora and large
 sequencing files are deliberately not stored in the repository.
 
 ## Releasing
@@ -481,6 +575,7 @@ MIT. See [LICENSE-BSD-3-CLAUSE] and [LICENSE-MIT].
 [`DecoderBuilder::input_page_size`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderBuilder.html#method.input_page_size
 [`DecoderBuilder::output_limit`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderBuilder.html#method.output_limit
 [`DecoderBuilder::expected_uncompressed_size`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderBuilder.html#method.expected_uncompressed_size
+[`DecoderBuilder::count_lines`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderBuilder.html#method.count_lines
 [`DecoderReader`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderReader.html
 [`DecoderStats::path`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderStats.html#structfield.path
 [`DecoderStats::configured_workers`]: https://docs.rs/rapidgzip-core/latest/rapidgzip_core/struct.DecoderStats.html#structfield.configured_workers

README.md

@@ -87,6 +87,19 @@ boundary or bypasses full member inflation and trailer verification.
 The AArch64 NEON implementation similarly guards each 16-byte input load and
 stores its comparison lanes only into a live local 16-byte array.
 
+## SIMD newline counting
+
+Optional line counting scans only final ordered byte slices. On x86-64, AVX2
+is called only after runtime detection; otherwise the SSE2 implementation is
+valid because SSE2 is part of the architecture baseline. Their unaligned loads
+are guarded by `offset + width <= bytes.len()`, and comparison masks are only
+counted, never used for memory access. On AArch64, Advanced SIMD is baseline and
+the NEON loop applies the same 16-byte bound before each unaligned load. Every
+implementation sends its shorter tail through safe scalar Rust and is
+differentially tested against that scalar implementation across vector edges,
+randomized contents, and every starting-address displacement from zero through
+63 bytes.
+
 ## Native DEFLATE bit loads
 
 The hot bit reader and Huffman peek perform an unaligned `u64` load only after