You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+59-1Lines changed: 59 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,13 +7,17 @@
7
7
8
8
Repository for Stochastic Optimization Solvers Benchmark implementation of the Window Sticker framework.
9
9
10
-
The benchmarking approach is described in this [preprint](https://arxiv.org/abs/2402.10255) titled: *Benchmarking the Operation of Quantum Heuristics and Ising Machines: Scoring Parameter Setting Strategies on Optimization Applications*.
10
+
The benchmarking approach is described in
11
+
[*Benchmarking the Operation of Quantum Heuristics and Ising Machines: Scoring Parameter Setting Strategies on Optimization Applications*](https://link.springer.com/article/10.1007/s42484-025-00311-2),
12
+
with the [arXiv preprint](https://arxiv.org/abs/2402.10255) also available.
11
13
12
14
Details of the implementation and an illustrative example for Wishart instances found [here](examples/wishart_n_50_alpha_0.5/wishart_n_50_alpha_0.50.ipynb) are given in this [document](stochastic-benchmarking-notes.pdf).
@@ -40,6 +44,60 @@ The current package implements the following functionality:
40
44
- Plot the Window sticker, comparing the performance curves corresponding to the virtual best, recommended parameters, and exploration-exploitation parameter setting strategies.
41
45
- Plots the values of the parameters and their best values with respect to the resource considered, a plot we call the Strategy plot. These plots can show the actual solver parameter values or the meta-parameters associated with parameter-setting strategies.
42
46
47
+
## Methodology Reliability
48
+
49
+
The Window Sticker workflow combines two different uncertainty questions that
50
+
should be interpreted separately. The cross-instance Window Sticker uncertainty
51
+
comes from train/test splits, interpolation, aggregation, and bootstrap
52
+
resampling across a problem family. It answers how a parameter-setting strategy
53
+
is expected to perform on unseen instances from that family. Per-instance
54
+
repeat-count reliability asks whether each solver, parameter setting, and
55
+
resource level has enough repeated stochastic runs to support its own success
56
+
probability, repeat count, and time-to-solution estimates.
57
+
58
+
Repeat reliability follows Noori et al. 2026:
59
+
60
+
Noori, Moslem, Elisabetta Valiante, Ignacio Rozada, Thomas Van Vaerenbergh, and
61
+
Masoud Mohseni. "[Statistical analysis for per-instance evaluation of stochastic
Physical Review Applied 25, no. 3 (2026): 034081. The related
64
+
[arXiv preprint](https://arxiv.org/abs/2503.16589) is titled "A Statistical
65
+
Analysis for Per-Instance Evaluation of Stochastic Optimizers: How Many Repeats
66
+
Are Enough?"
67
+
68
+
The analytic guarantees implemented here apply to Bernoulli success events and
69
+
metrics derived from their success probabilities: `R_c`, RTT/TTS, CETS, and
70
+
thresholded continuous metrics. For example, a continuous energy, Response, or
71
+
PerfRatio value can be analyzed with repeat reliability only after the workflow
72
+
defines a threshold that turns each run into success or failure.
73
+
74
+
The original Window Sticker bootstrap remains the uncertainty model for
75
+
continuous Response curves and for continuous PerfRatio curves that are not
76
+
converted to success thresholds. Those bootstrap intervals are useful for
77
+
cross-instance benchmarking, but they are not a substitute for the analytic
78
+
repeat-count checks above.
79
+
80
+
Repeat reliability addresses several methodology criticisms directly:
81
+
82
+
| Criticism | Documentation and package response |
83
+
| --- | --- |
84
+
| Repeat-count sufficiency | Report `required_trials`, `additional_trials_required`, `reliable`, and `reliability_status` so users can tell whether more stochastic runs are needed. |
85
+
| Bootstrap-only uncertainty | Keep bootstrap intervals for cross-instance continuous curves, and use analytic Bernoulli intervals for per-instance success probabilities and derived repeat-count metrics. |
86
+
| Noisy HPO choices | Surface repeat reliability before treating a parameter choice as stable, especially when success rates are small or intervals are wide. |
87
+
| CI-overlap ambiguity | Flag `ci_overlaps_best` and `statistically_unresolved` comparisons instead of implying that overlapping intervals identify a clear winner. |
88
+
| Virtual-best optimism | Document virtual best as an optimistic, unattainable reference and pair it with reliability checks when judging whether observed gaps are meaningful. |
89
+
90
+
Use [docs/passive_repeat_reliability_reports.md](docs/passive_repeat_reliability_reports.md)
91
+
to add passive repeat-reliability reports to existing benchmark data. Use
Copy file name to clipboardExpand all lines: examples/general_workflow.md
+59-3Lines changed: 59 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,9 +8,15 @@ Given a benchmark task, the workflow generally follows these steps:
8
8
4. Train/Test split and Statistical Aggregation
9
9
5. Virtual Best Baseline
10
10
6. Evaluating Parameter Recommendation Strategies
11
-
7. Visualization
11
+
7. Repeat Reliability
12
+
8. Visualization
13
+
14
+
Each step is explored in greater detail along with code implementation at
15
+
`examples/QAOA_iterative/qaoa_demo.ipynb`. For the QAOA benchmark context, see
16
+
the published ACM Transactions on Quantum Computing article
17
+
[*Optimization Applications as Quantum Performance Benchmarks*](https://doi.org/10.1145/3678184)
18
+
and its [arXiv preprint](https://arxiv.org/abs/2302.02278).
12
19
13
-
Each step is explored in greater detail along with code implementation at `examples/QAOA_iterative/qaoa_demo.ipynb`.
14
20
## Define the Metrics
15
21
In this first step, we configure the central object for the benchmark task. We define:
16
22
@@ -41,9 +47,59 @@ After having devised parameter recommendations from the training set, we need a
41
47
42
48
1. We can aggregate statistics across all training instances to learn a parameter recipe. We refer to this as the Aggregate-then-Recommend projection strategy.
43
49
2. It is also possible to look at what parameters work best for each instance individually. After this, we can average those recommendations.
50
+
51
+
## Repeat Reliability
52
+
53
+
Noori et al. 2026 show that per-instance stochastic optimizer conclusions can
54
+
be unreliable when they are based on too few repeats:
55
+
56
+
Noori, Moslem, Elisabetta Valiante, Ignacio Rozada, Thomas Van Vaerenbergh, and
57
+
Masoud Mohseni. "[Statistical analysis for per-instance evaluation of stochastic
Physical Review Applied 25, no. 3 (2026): 034081. The related
60
+
[arXiv preprint](https://arxiv.org/abs/2503.16589) is titled "A Statistical
61
+
Analysis for Per-Instance Evaluation of Stochastic Optimizers: How Many Repeats
62
+
Are Enough?"
63
+
64
+
This check is separate from the cross-instance Window Sticker uncertainty used
65
+
elsewhere in the workflow. Window Sticker plots summarize expected performance
66
+
over unseen instances from a problem family. Repeat reliability checks whether a
67
+
specific instance, solver configuration, and resource level has enough repeated
68
+
runs to estimate its success probability and the derived repeat counts.
69
+
70
+
Analytic repeat-reliability guarantees apply when each run can be treated as a
71
+
Bernoulli success event. In this package that covers:
72
+
73
+
-`R_c`, the repeats required to reach a target success confidence
74
+
- RTT/TTS values derived from `R_c` and runtime-per-repeat scaling
75
+
- CETS values derived from `R_c`, iterations, and effort-per-iteration scaling
76
+
- Thresholded continuous metrics, such as Response or PerfRatio after a
77
+
success threshold converts each run into success or failure
78
+
79
+
Continuous Response curves and continuous PerfRatio curves remain
80
+
bootstrap-based unless they are converted to thresholded success events. Their
81
+
bootstrap intervals describe empirical cross-instance variation and resampling
82
+
uncertainty, not the analytic repeat-count guarantees from the paper.
83
+
84
+
Use `repeat_reliability_report` or `stochastic_benchmark.run_RepeatReliability`
85
+
before treating a benchmark conclusion as final. The resulting
86
+
`required_trials`, `additional_trials_required`, `reliable`, and
87
+
`reliability_status` columns tell users whether the existing data are reliable
88
+
enough or whether the solver should be rerun.
89
+
90
+
## Methodology Criticism Checklist
91
+
92
+
| Criticism | Workflow response |
93
+
| --- | --- |
94
+
| Repeat-count sufficiency | Check required and additional trials before trusting a per-instance success estimate. |
95
+
| Bootstrap-only uncertainty | Use analytic Bernoulli intervals for success events and keep bootstrap intervals for continuous cross-instance curves. |
96
+
| Noisy HPO choices | Treat parameter recommendations as provisional when repeat reliability is low or intervals are wide. |
97
+
| CI-overlap ambiguity | Mark overlapping comparisons as statistically unresolved instead of selecting a winner from point estimates alone. |
98
+
| Virtual-best optimism | Interpret virtual best as an unattainable optimistic reference, not as a deployable strategy. |
99
+
44
100
## Visualization
45
101
46
102
The framework's results can be visualized, mainly, through two lenses:
47
103
48
104
1. We can look at how our defined metric of interest compares against the defined resource. Here we can compare the Virtual Best, the Projection from the Training Set, and the Performance from the Training Set. Essentially, this tells us how close we can expect to be from the Virtual Best when new problem instances come in.
49
-
2. We can look at how we should distribute our input parameters for different resource amounts. Getting back to the previous **energy** example, if we have a given amount we are willing to spend, this can be achieved via different combinations of inputs. This analysis tells us what inputs to pick in order to achieve the performance metrics computed by the framework.
105
+
2. We can look at how we should distribute our input parameters for different resource amounts. Getting back to the previous **energy** example, if we have a given amount we are willing to spend, this can be achieved via different combinations of inputs. This analysis tells us what inputs to pick in order to achieve the performance metrics computed by the framework.
0 commit comments