Skip to content

Document repeat reliability methodology - #81

Merged
bernalde merged 2 commits into
usra-riacs:mainfrom
bernalde:fix/issue-76-methodology-docs
May 27, 2026
Merged

Document repeat reliability methodology#81
bernalde merged 2 commits into
usra-riacs:mainfrom
bernalde:fix/issue-76-methodology-docs

Conversation

@bernalde

Copy link
Copy Markdown
Collaborator

Summary

  • Document how repeat reliability changes Window Sticker methodology claims.
  • Separate cross-instance bootstrap uncertainty from per-instance repeat-count reliability.
  • Clarify that Noori et al. 2026 analytic guarantees apply to Bernoulli success events, R_c, RTT/TTS, CETS, and thresholded continuous metrics, while continuous Response and PerfRatio curves remain bootstrap-based unless thresholded.
  • Add a methodology criticism mapping and documentation regression tests.

Tests run

  • python -m pytest tests/test_methodology_docs.py -q - 2 passed.
  • python run_tests.py unit --fast - 413 passed, 12 warnings.
  • flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics - 0 critical issues.
  • flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics - exited 0; printed existing advisory style warnings.
  • python run_tests.py integration - 11 passed.

Notes about tests not run

  • Did not run local tutorial notebook execution or local coverage report generation; CI covers those heavier checks.
  • No separate type-check or formatting command is documented for this repository.

Closes #76

@codecov

codecov Bot commented May 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@bernalde bernalde left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

  1. Blocking issues: none.
  2. Nonblocking issues: none.
  3. Questions: none.
  4. Tests run and outcomes:
    • python -m pytest tests/test_methodology_docs.py -q - 2 passed.
    • python run_tests.py all --fast - unit, integration, and smoke suites passed.
    • flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics - 0 critical issues.
    • flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics - exited 0 with existing advisory style warnings.
    • gh pr checks 81 --repo usra-riacs/stochastic-benchmark - all reported checks passed.
  5. Merge recommendation: merge-ready. I am submitting this as COMMENT because GitHub does not allow the authenticated author account to approve its own PR.

@bernalde
bernalde merged commit 91d053c into usra-riacs:main May 27, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document methodology changes and criticisms addressed by repeat reliability

1 participant