Skip to content

docs(cats): false-positive rule numbering has drifted (70 rules, comments say 48/49, 21 unnumbered) #667

Description

@ericfitz

Summary

test/cats/false-positives.yaml now holds 70 rules. The documentation and the in-file comments
still describe 48.

test/cats/README.md states the numbering is a maintenance obligation:

Each rule's leading comment records its actual file position as rule N of 48; if you add,
remove, or reorder a rule, renumber every comment (and the total) to match.

That has not been happening.

Current state

rules in the file 70
rules carrying a rule N of M comment 49
rules with no numbered comment 21
distinct denominators in use 48 and 49
highest N present 49

CLAUDE.md and test/cats/README.md both say "48 rules". cats doctor reports 70.

Why it matters

Rule order is load-bearing — evaluation is first-match-wins, and the README is explicit that
inserting a rule in the wrong place "can silently change which rule (or whether any rule) suppresses
a given fuzzer finding". The position comments exist so a reader can tell where a rule sits without
counting. With 21 rules unnumbered and two different totals in circulation, they no longer do that,
and a reader who trusts them is worse off than one who ignores them.

Acceptance Criteria

  • Renumber every rule comment to its real position, with a single correct total
  • Update the counts in test/cats/README.md and CLAUDE.md
  • Decide whether hand-maintained position comments are worth keeping. A check in
    make lint, or a cats subcommand that prints rules with their evaluation order, would give
    the same information without a manual step that has already failed twice
  • If the comments are kept, add a check that fails when they disagree with actual position

Notes

Noticed while confirming that no existing rule masks the threats/bulk fix in #658 (none does).
Not fixed inline there: renumbering 70 entries is mechanical churn in a file where order matters,
and it does not belong in a change that is already large.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions