Skip to content

Commit 897f9a8

Browse files
codexByron
andcommitted
Add backend regression benchmark harness
Add known-good backend regression benchmark harness Introduce an ignored integration test that measures backend-sensitive compress/decompress cases against a known-good commit. - the test benchmarks the current checkout in release mode - it creates a detached worktree at KNOWN_GOOD_COMMIT on the same runner - it generates a tiny temporary Cargo project whose manifest points `flate2` at that worktree - it copies `tests/support/backend-regression-driver.rs` into that project and runs it to produce baseline CSV data - it compares the current results against those baseline measurements and fails when the observed slowdown exceeds the combined measurement uncertainty of the current run and the baseline run - run a single backend locally with: `cargo test --release --test backend-regression-bench -- --ignored --exact backend_regression_bench --nocapture` - switch backends the same way the rest of the test matrix does, for example: `cargo test --release --test backend-regression-bench --features zlib-rs --no-default-features -- --ignored --exact backend_regression_bench --nocapture` - optionally, inspect `target/backend-bench/*.csv` for the raw baseline/current measurements after a run This is intended to be maintained as a regression guard: - update KNOWN_GOOD_COMMIT only when intentionally accepting a new performance baseline - keep the driver source reusable and branch-independent so the baseline setup stays lightweight and easy to reason about ``` > Add this benchmark (#544 (comment)) to version of the library before #502 was merged to get a baseline for all backends. Store that baseline, and add a CI job that checks against the baseline. Consider other uses of performance-critical low-level methods and see if it makes sense to add more kinds of tests to the baseline the most recent version. Success means that the baseline as run on main reproduces issue #544 as well, there is no need to attempt to fix it. Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>
1 parent e4429c4 commit 897f9a8

3 files changed

Lines changed: 944 additions & 0 deletions

File tree

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Backend performance regression
2+
3+
on:
4+
push:
5+
pull_request:
6+
workflow_dispatch:
7+
8+
jobs:
9+
backend-regression:
10+
name: Backend regression (${{ matrix.backend.name }})
11+
runs-on: ubuntu-latest
12+
permissions:
13+
contents: read
14+
strategy:
15+
fail-fast: false
16+
matrix:
17+
backend:
18+
- name: rust_backend
19+
command: cargo test --release --test backend-regression-bench -- --ignored --nocapture
20+
- name: zlib-rs
21+
command: cargo test --release --test backend-regression-bench --features zlib-rs --no-default-features -- --ignored --nocapture
22+
- name: zlib
23+
command: cargo test --release --test backend-regression-bench --features zlib --no-default-features -- --ignored --nocapture
24+
- name: zlib-ng
25+
command: cargo test --release --test backend-regression-bench --features zlib-ng --no-default-features -- --ignored --nocapture
26+
- name: zlib-ng-compat
27+
command: cargo test --release --test backend-regression-bench --features zlib-ng-compat --no-default-features -- --ignored --nocapture
28+
steps:
29+
- uses: actions/checkout@v4
30+
with:
31+
fetch-depth: 0
32+
- name: Install Rust
33+
run: rustup update stable --no-self-update && rustup default stable
34+
shell: bash
35+
- name: Run backend regression benchmark
36+
run: ${{ matrix.backend.command }}
37+
- name: Upload backend benchmark CSV
38+
if: always()
39+
uses: actions/upload-artifact@v4
40+
with:
41+
name: backend-regression-${{ matrix.backend.name }}
42+
path: target/backend-bench/${{ matrix.backend.name }}*.csv

0 commit comments

Comments
 (0)