Run SA tutorial reliability analysis

[bernalde]

Summary

• Add Simulated Annealing repeat-reliability helpers that derive configurable success thresholds from the tutorial's best/random performance-ratio convention.
• Add a focused companion notebook, examples/Simulated_Annealing/reliability_analysis.ipynb, that runs the repeat-reliability workflow as a reusable example.
• Register the companion notebook in the tutorial manifest and load its example data from the tracked results/all_raw_runs.pkl aggregate so it executes in fresh checkouts.
• Add focused tests for thresholding, diagnostic selection, aggregate raw-run loading, and the tutorial manifest entries.

Closes #75

Tests run

• jq empty examples/Simulated_Annealing/reliability_analysis.ipynb examples/tutorials.json - passed.
• python -m pytest tests/test_verify_tutorials.py tests/test_simulated_annealing_reliability.py - passed, 14 tests.
• python scripts/verify_tutorials.py --manifest /tmp/pr80-sa-companion-tutorial.json --output-dir /tmp/pr80-sa-companion-executed-v2 - passed; executed the companion reliability notebook.
• python run_tests.py unit - passed, 408 tests, 12 warnings.
• python scripts/verify_tutorials.py --output-dir /tmp/pr80-full-tutorials-with-companion-v2 - passed; executed all runnable self-contained tutorial notebooks and skipped the configured slow notebooks.
• git diff --check - passed.

Notes

• python run_tests.py coverage was not run locally to avoid generating coverage artifacts; CI coverage jobs cover that path.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[bernalde]

1. Blocking issues: none.
2. Nonblocking issues: diagnostics drop target-specific repeat-count columns when target_confidence is customized.
3. Questions: none.
4. Tests run and outcomes:

• python -m pytest tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py: passed, 11 tests.
• python run_tests.py unit: passed, 405 tests, 12 warnings.
• python run_tests.py integration: passed, 11 tests.
• python scripts/verify_tutorials.py --output-dir /tmp/pr80-executed-notebooks: initial sandboxed run failed before notebook execution because Jupyter could not open local kernel sockets (PermissionError: [Errno 1] Operation not permitted); reran outside the sandbox and it passed, including the modified Simulated Annealing notebook.
• flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics: passed, reported 0.
• flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics: exited 0 and reported existing advisory style findings.
• flake8 examples/Simulated_Annealing/reliability_analysis.py tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py --count --select=E9,F63,F7,F82 --show-source --statistics: passed, reported 0.
• git diff --check upstream/main...HEAD: passed.

5. Merge as-is: yes, I did not find any blocking issues. I am submitting this as COMMENT because the authenticated GitHub account is the PR author and GitHub does not allow approving your own PR.

[bernalde]

Follow-up pushed for the unresolved review thread.

Commits pushed:

• dba53a806a950e0f190ebfa8665f23feadc60a34 Preserve custom repeat diagnostics (Run reliability analysis through the Simulated Annealing tutorial #75)

Main changes:

• select_reliability_diagnostics now keeps neutral R_c estimate/interval columns.
• The diagnostic selector also dynamically includes whichever target-specific repeat alias exists in the report, such as R99 or R95.
• Added a regression test covering target_confidence=0.95 so custom repeat-count aliases are preserved.

Tests run:

• python -m pytest tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py passed, 12 tests.
• flake8 examples/Simulated_Annealing/reliability_analysis.py tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py --count --select=E9,F63,F7,F82 --show-source --statistics passed, reported 0.
• git diff --check passed.
• python run_tests.py unit passed, 406 tests, 12 warnings.
• python scripts/verify_tutorials.py --manifest /tmp/pr80-sa-tutorials.json --output-dir /tmp/pr80-sa-executed-notebooks passed for the Simulated Annealing tutorial.

Comments intentionally not addressed:

• The Codecov top-level report is informational and already reports modified coverable lines covered.

Remaining risks or follow-up items:

• None identified.

[bernalde]

1. Blocking issues

• The inline comment on select_reliability_diagnostics should be addressed before merge.

2. Nonblocking issues

• None.

3. Questions

• None.

4. Tests run and outcomes

• python -m pytest tests/test_verify_tutorials.py tests/test_simulated_annealing_reliability.py passed, 14 tests.
• python run_tests.py unit passed, 408 tests, 12 warnings.
• python scripts/verify_tutorials.py --output-dir /tmp/pr80-review-executed-notebooks passed.
• flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics passed with 0.
• flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics exited 0 with existing advisory style findings in src/.
• git diff --check main...HEAD passed.
• gh pr checks 80 --repo usra-riacs/stochastic-benchmark reports all checks passing.

5. Merge recommendation

• I would not merge this until the blocking issue above is addressed.

[bernalde]

Follow-up pushed for the current review comments.

Commits pushed:

• 5090f3ede9ec3699980398d83fb5ef2bc8196be5 Keep decimal SA repeat diagnostics (Run reliability analysis through the Simulated Annealing tutorial #75)

Main changes:

• Updated the SA reliability diagnostic column matcher to keep target-specific repeat aliases with decimal confidence labels, such as R97_5.
• Extended the custom-confidence regression test to cover both target_confidence=0.95 and target_confidence=0.975.

Tests run:

• python -m pytest tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py passed, 15 tests.
• flake8 examples/Simulated_Annealing/reliability_analysis.py tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py --count --select=E9,F63,F7,F82 --show-source --statistics passed, reported 0.
• git diff --check passed.
• python run_tests.py unit passed, 409 tests, 12 warnings.
• python scripts/verify_tutorials.py --output-dir /tmp/pr80-comments-executed-notebooks passed.

Comments intentionally not addressed:

• The earlier unresolved R95 inline thread was already addressed by dba53a806a950e0f190ebfa8665f23feadc60a34; no additional code change was needed for that thread.
• The Codecov top-level comment is informational and already reports modified coverable lines covered.
• The prior follow-up/status comments are informational and do not require code changes.

Remaining risks or follow-up items:

• None identified.

[bernalde]

1. Blocking issues

• None.

2. Nonblocking issues

• None.

3. Questions

• None.

4. Tests run and outcomes

• python -m pytest tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py passed, 15 tests.
• flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics passed with 0.
• flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics exited 0 and reported existing advisory style findings in src/, matching CI's non-blocking configuration.
• git diff --check upstream/main...HEAD passed.
• python run_tests.py unit passed, 409 tests, 12 warnings.
• python run_tests.py integration passed, 11 tests.
• python scripts/verify_tutorials.py --output-dir /tmp/pr80-review-final-executed-notebooks passed.
• gh pr checks 80 --repo usra-riacs/stochastic-benchmark reports all checks passing.

5. Merge recommendation

• The PR should be merged as-is. I am submitting this as COMMENT because the authenticated GitHub account is the PR author, and GitHub does not allow approving your own PR.

[bernalde]

Review summary:

1. Blocking issues: none.
2. Nonblocking issues: none requiring inline comments.
3. Questions: none.
4. Tests run and outcomes:
   • python -m pytest tests/test_simulated_annealing_reliability.py tests/test_verify_tutorials.py -q - passed, 17 passed.
   • python run_tests.py unit - passed, 411 passed, 12 warnings.
   • PYTHONPATH=/home/bernalde/repos/stochastic-benchmark/src MPLCONFIGDIR=/tmp/matplotlib python scripts/verify_tutorials.py --manifest /tmp/pr80-review-manifest.json --output-dir /tmp/pr80-review-executed --timeout 600 - passed when rerun outside the sandbox. The first sandboxed attempt failed because Jupyter could not create local kernel sockets (PermissionError: [Errno 1] Operation not permitted).
   • flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics - passed, output 0.
   • flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics - completed successfully with advisory warnings in existing src files.
   • git diff --check origin/main...HEAD - passed.
   • gh pr checks 80 --repo usra-riacs/stochastic-benchmark - all reported checks pass: codecov/patch, coverage-summary, integration-tests, test (3.10), test (3.11), test (3.12), and tutorial-smoke.
5. Merge as-is: yes from a maintainer standpoint. I am submitting this as COMMENT rather than APPROVE because the authenticated GitHub account is the PR author.