QAOA parameter setting analysis using IBM Quantum data - #50
Conversation
- Expand IBM_QAOA patterns to all examples subdirectories - Add general artifact patterns (plots, checkpoints, progress) - Add data file patterns (pkl, npz, npy) - Ignore accidentally created repo root directory
Add comprehensive test fixtures for IBM QAOA data processing: - 4 real experimental JSON files with varied instance/depth configurations - 4 synthetic edge case fixtures (multi-trial, missing fields, empty) - README documenting schema, boundaries, and usage patterns Fixtures support testing of: - Single vs multi-trial bootstrap behavior - IBM-specific filename parsing - Missing field handling - Empty/malformed data edge cases
Add 32 tests organized in 6 test classes covering: Unit Tests: - QAOAResult dataclass creation - parse_qaoa_trial with various data formats - load_qaoa_results with missing/malformed data - convert_to_dataframe transformations - group_name_fcn filename parsing - prepare_stochastic_benchmark_data pickle I/O Integration Tests: - process_qaoa_data end-to-end pipeline - GTMinEnergy injection for missing ground truth - Single-trial bootstrap fabrication - Interpolation fallback behavior - Train/test split generation Edge Cases: - Missing trainer information - Missing optimal parameters - Empty trials list - Multi-trial synthetic data All tests use fixtures and proper mocking for multiprocessing. Test coverage validates IBM-specific logic boundaries.
Implement 4 high-impact optimizations for ~1K file scale: 1. ProcessingConfig dataclass for centralized configuration: - persist_raw: gate pickle writes during ingestion - interpolate_diversity_threshold: diversity-based interpolation - fabricate_single_trial: control single-trial bootstrap - seed: reproducible train/test splits - log_progress_interval: configurable progress logging 2. Structured logging infrastructure: - Replace print statements with logging module - Add progress logging every N files - Proper INFO/WARNING levels for errors - Timestamps and levels for production observability 3. In-memory aggregation with conditional pickle persistence: - persist_raw=True: write pickles to exp_raw/ subdirectory - persist_raw=False: aggregate in memory, skip ingestion pickles - Generate temporary pickles only when needed for bootstrap - Expected 1-2s savings for 1K files when disabled 4. Diversity-based interpolation heuristic: - Replace row count (n_rows <= 5) with diversity metric - diversity = unique_instances × unique_depths - Skip interpolation when diversity < threshold - Prevents spurious skips on sparse but valid grids Additional improvements: - Add try/except for malformed JSON files with warnings - Use config.seed for reproducible train/test splits - Fix pickle paths to use exp_raw subdirectory convention - Add enumeration to ingestion loop for progress tracking All changes maintain backward compatibility with default config. Expected performance improvement: ~15s for 1K files (from ~20-25s).
Add comprehensive performance optimization guide: Phase 1 - Implemented (4 changes): - ProcessingConfig dataclass for centralized configuration - Structured logging infrastructure - In-memory aggregation with persist_raw flag - Diversity-based interpolation heuristic Phase 2 - Deferred Enhancements (6 optimizations): 1. Parallel I/O with ThreadPoolExecutor (3-5x potential speedup) 2. Parquet output format (faster writes, smaller files) 3. orjson for JSON parsing (~2x speedup) 4. Lazy bootstrap fabrication (skip unnecessary computation) 5. Categorical dtypes for memory efficiency 6. Rich diversity metrics (entropy-based quality assessment) Each enhancement documented with: - Problem/solution description - Expected impact and thresholds - Implementation complexity - Testing requirements Target metrics: - Phase 1: <15s for 1K files (from ~20-25s baseline) - Phase 2: <10s with parallelization - Scale guidance: When to apply each optimization
Clear execution outputs and intermediate results to reduce repo size. Notebook structure and analysis code preserved.
Add ibm_qaoa_analysis_hardware.ipynb for analyzing real quantum hardware results from IBM systems. Complements simulation analysis with hardware- specific metrics and comparisons.
Move all pandas-specific test files from tests/Pandas_Group_Tests/ to tests/: - test_interpolate_pandas.py - test_stats_pandas.py - test_stochastic_benchmark_pandas.py - test_training_pandas.py Remove empty Pandas_Group_Tests subdirectory for better test organization. All 13 tests still passing after move.
- Update processing script to detect three optimization states: 'opt', 'noOpt', and None - Add marker differentiation in plotting: circles for opt, x for noOpt, squares for no flag - Use depth-specific colors for all marker types with appropriate legend labels - Extract optimization flag from filename patterns (_opt_, _noOpt_, or neither) - Fallback to Energy metric when Approximation Ratio not available in JSON
- Load minmax cuts from JSON files in R3R/minmax_cuts directory - Add maxcut_approximation_ratio() function using formula: cut_val = energy + 0.5 * sum_weights approx_ratio = (cut_val - min_cut) / (max_cut - min_cut) - Update convert_to_dataframe() to use calculated approximation ratios - Update process_qaoa_data() to load and pass minmax data - Add proper error handling and validation for edge cases
- Test minmax cuts loading from directory - Test approximation ratio calculation - Test end-to-end processing with minmax integration - Verify non-NaN approximation ratios in output
…d comparison - Update data loading to use 'optimized' column from process_qaoa_data - Create separate method names (FA_opt, FA_noOpt, TQA_opt, TQA_noOpt) - Update methods_to_compare list to include all 7 method variants - Change legends to single-column layout for better readability - Invalidate cache to force reprocessing with new method names - All variants treated independently in statistical analysis and rankings
|
@anurag-r20 I've opened a new pull request, #51, to work on those changes. Once the pull request is ready, I'll request review from you. |
Co-authored-by: anurag-r20 <68232146+anurag-r20@users.noreply.github.com>
PR review update: fix failing tests, surfaced actionable failures
…ing_IBM # Conflicts: # .gitignore # examples/QAOA_iterative/qaoa_demo.ipynb # src/stochastic_benchmark.py
bernalde
left a comment
bernalde
left a comment
There was a problem hiding this comment.
Blocking issues:\n- The documented unit suite currently fails in the new pandas stochastic-benchmark tests because the fixture passes a stochastic_benchmark instance where the experiment classes require the ExperimentParameters contract, including checkpoint_path.\n- The new IBM_QAOA processing modules add a large user-facing parsing/analysis surface without active collected tests for the current modules.\n\nNonblocking issues:\n- The pandas groupby.apply compatibility changes reintroduce FutureWarnings under the repo's required pandas 2.3.x and should be made future-safe.\n- The IBM_QAOA script defaults still point at contributor-local absolute paths, which makes the documented workflow hard to reproduce outside that machine.\n\nQuestions:\n- Should the IBM_QAOA workflow add an example-specific requirements/setup section for qiskit, qiskit-aer, qaoa_training_pipeline, and the QAOA-Parameter-Setting checkout?\n\nTests run and outcomes:\n- python -m pytest tests/test_interpolate_pandas.py tests/test_stats_pandas.py tests/test_stochastic_benchmark_pandas.py tests/test_training_pandas.py tests/test_df_utils.py tests/test_stats.py -q on the base Python 3.13: collection failed because the base env lacks multiprocess; reran in the documented CI conda env.\n- conda run -n stochastic-benchmark-ci-py310 python -m pytest tests/test_interpolate_pandas.py tests/test_stats_pandas.py tests/test_stochastic_benchmark_pandas.py tests/test_training_pandas.py tests/test_df_utils.py tests/test_stats.py -q: 37 passed, 6 failed, all in tests/test_stochastic_benchmark_pandas.py.\n- conda run -n stochastic-benchmark-ci-py310 python run_tests.py unit: 217 passed, 6 failed, same failures.\n- conda run -n stochastic-benchmark-ci-py310 python run_tests.py integration: 11 passed.\n- conda run -n stochastic-benchmark-ci-py310 python -m py_compile examples/IBM_QAOA/run_prepare_pss_campaign.py examples/IBM_QAOA/src/Processing.py examples/IBM_QAOA/src/approx_ratio_calc.py examples/IBM_QAOA/src/simulation_validation.py examples/IBM_QAOA/src/utils.py: passed.\n- conda run -n stochastic-benchmark-ci-py310 python -c \"import sys; sys.path.insert(0, 'examples/IBM_QAOA'); import src.Processing, src.approx_ratio_calc, src.utils, src.simulation_validation; print('imports ok')\": passed.\n- conda run -n stochastic-benchmark-ci-py310 python examples/IBM_QAOA/run_prepare_pss_campaign.py --help: passed.\n\nThe PR should not be merged as-is. I would not merge this until the blocking issues above are addressed.
| sb.stat_params = StatsParameters(metrics=['response'], stats_measures=[Median()]) | ||
|
|
||
| # RESTORED: Global setup for 'here' | ||
| sb.here = type('obj', (object,), {'checkpoints': '/tmp'}) |
There was a problem hiding this comment.
Blocking: This fixture constructs a stochastic_benchmark object but the tests pass it directly to VirtualBestBaseline and ProjectionExperiment. Those classes expect the experiment-parameter contract, including checkpoint_path, so python run_tests.py unit fails six tests with AttributeError: 'stochastic_benchmark' object has no attribute 'checkpoint_path'. Build an experiments.ExperimentParameters fixture, or call sb.get_experiment_parameters() after populating the required state, and use tmp_path for the checkpoint directory so these tests exercise the production object shape.
| @@ -0,0 +1,954 @@ | |||
| ''' This file is for processing QAOA data files generated by IBM using QAOA. | |||
There was a problem hiding this comment.
Blocking: This new IBM QAOA processing surface is not covered by active collected tests. The PR adds fixtures under tests/fixtures/ibm_qaoa, but the comprehensive processing tests are under examples/IBM_QAOA/Archive/ibm_qaoa_processing_tests.py, which pytest does not collect and which imports the old ibm_qaoa_processing module path. Please move focused tests into tests/test_ibm_qaoa_processing.py for the current modules, including QAOAHardware.load_hardware_instance, QAOATraining.load_training_instance, min/max cut lookup, approximation-ratio conversion, and training-cost resolution.
| @@ -108,7 +108,7 @@ def br(df): | |||
| smooth, | |||
| ) | |||
|
|
|||
| vb = df.groupby(groupby).apply(br, include_groups=False).reset_index() | |||
| vb = df.groupby(groupby).apply(br).reset_index() # include_groups=False, Pandas Version Error | |||
There was a problem hiding this comment.
Nonblocking: Removing include_groups=False brings back the pandas DataFrameGroupBy.apply operated on the grouping columns FutureWarning under the repo's required pandas 2.3.x, and pandas will change this behavior in a future release. Since requirements.txt already requires pandas>=2.3, please restore include_groups=False here and in the parallel interpolate.py/stats.py groupby calls, or add a small compatibility helper if older pandas support is still needed.
| ) | ||
|
|
||
|
|
||
| DEFAULT_MAIN_REPO = Path("/mnt/c/Users/rames102/Desktop/QAOA-Parameter-Setting") |
There was a problem hiding this comment.
Nonblocking: This default points to a contributor-specific desktop path, so the script/notebook workflow leaves the documented path for any other checkout unless the user already knows which env var or CLI flag to override. Prefer no personal absolute default: derive paths from repo-relative locations where possible, or make the argument required and document QAOA_PARAMETER_SETTING_ROOT and QAOA_TRAINING_PIPELINE_ROOT in the notebook README.
|
|
||
| ## Notes | ||
|
|
||
| - The notebook expects local data directories/instance directories to be available (paths are currently set inside the notebook). |
There was a problem hiding this comment.
Question: The new notebooks/scripts require external projects and packages (qaoa_training_pipeline, qiskit, qiskit-aer, and the QAOA-Parameter-Setting data checkout), but the repo's requirements-examples.txt only installs scikit-learn and this README does not give setup commands. Should this workflow add an IBM_QAOA-specific requirements file or a setup section with the required env vars and dependency installation?
bernalde
left a comment
bernalde
left a comment
There was a problem hiding this comment.
Blocking issues:
- The existing
tests/test_stochastic_benchmark_pandas.pyblocking thread still applies. The full documented test suite fails six newly added tests because they pass astochastic_benchmarkinstance directly toVirtualBestBaselineandProjectionExperiment, while those classes require the experiment-parameter contract withcheckpoint_path. - The existing IBM QAOA processing coverage blocking thread still applies. The PR adds fixtures and documents
tests/test_ibm_qaoa_processing.py, but that file is absent; the only comprehensive processing tests are archived, uncollected, and currently fail during collection. - The PR is currently merge-conflicting with
main(mergeable: CONFLICTING), so it cannot be merged as-is.
Nonblocking issues:
- The existing
include_groups=Falsethread still applies. The PR reintroduces pandas groupby warnings under the required pandas 2.3.x range and should either restoreinclude_groups=Falseor add an explicit compatibility helper. - The existing personal absolute path/defaults comment still applies for the IBM QAOA workflow.
Questions:
- The existing setup/dependency question for the IBM QAOA notebooks/scripts remains open.
Tests run and outcomes:
conda run -n stochastic-benchmark-ci-py310 python -m pytest tests/ -q: failed, 6 failed and 228 passed. All failures are intests/test_stochastic_benchmark_pandas.py.conda run -n stochastic-benchmark-ci-py310 python -m pytest tests/test_interpolate_pandas.py tests/test_stats_pandas.py tests/test_stochastic_benchmark_pandas.py tests/test_training_pandas.py tests/test_df_utils.py tests/test_stats.py -q: failed, 6 failed and 37 passed. Samecheckpoint_pathfailures.conda run -n stochastic-benchmark-ci-py310 python -m pytest tests/test_interpolate.py tests/test_training.py tests/test_stats.py -q: passed, 45 passed.conda run -n stochastic-benchmark-ci-py310 python -m pytest examples/IBM_QAOA/Archive/ibm_qaoa_processing_tests.py -q: failed during collection withNameError: name 'Dict' is not defined, and this path is outside configured pytest collection.conda run -n stochastic-benchmark-ci-py310 flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics: passed, 0 critical findings.conda run -n stochastic-benchmark-ci-py310 flake8 src --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics: exited 0 with 142 reported style warnings, consistent with the command's warning-only behavior.gh pr checks 50 --repo usra-riacs/stochastic-benchmark: no checks reported on the PR branch.
I would not merge this until the blocking issues above are addressed.
…ing_IBM # Conflicts: # src/plotting.py # src/stats.py
|
@bernalde All checks passed, Added documentation for dependencies |
Summary
This PR adds adds a QAOA parameter setting analysis workflow using IBM Quantum data,
How to Run Experiments
examples/IBM_QAOA/notebooks/Analysis.ipynbNotes