Commit 897f9a8
Add backend regression benchmark harness
Add known-good backend regression benchmark harness
Introduce an ignored integration test that measures backend-sensitive
compress/decompress cases against a known-good commit.
- the test benchmarks the current checkout in release mode
- it creates a detached worktree at KNOWN_GOOD_COMMIT on the same runner
- it generates a tiny temporary Cargo project whose manifest points
`flate2` at that worktree
- it copies `tests/support/backend-regression-driver.rs` into that
project and runs it to produce baseline CSV data
- it compares the current results against those baseline measurements and
fails when the observed slowdown exceeds the combined measurement
uncertainty of the current run and the baseline run
- run a single backend locally with:
`cargo test --release --test backend-regression-bench -- --ignored --exact backend_regression_bench --nocapture`
- switch backends the same way the rest of the test matrix does, for
example:
`cargo test --release --test backend-regression-bench --features zlib-rs --no-default-features -- --ignored --exact backend_regression_bench --nocapture`
- optionally, inspect `target/backend-bench/*.csv` for the raw baseline/current
measurements after a run
This is intended to be maintained as a regression guard:
- update KNOWN_GOOD_COMMIT only when intentionally accepting a new
performance baseline
- keep the driver source reusable and branch-independent so the baseline
setup stays lightweight and easy to reason about
```
> Add this benchmark (#544 (comment)) to version of the library before #502 was merged to get a baseline for all backends. Store that baseline, and add a CI job that checks against the baseline.
Consider other uses of performance-critical low-level methods and see if it makes sense to add more kinds of tests to the baseline the most recent version.
Success means that the baseline as run on main reproduces issue #544 as well, there is no need to attempt to fix it.
Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>1 parent e4429c4 commit 897f9a8
3 files changed
Lines changed: 944 additions & 0 deletions
File tree
- .github/workflows
- tests
- support
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
0 commit comments