Skip to content

ci-evidence.md

Latest commit

 

History

History
180 lines (137 loc) · 7.43 KB

ci-evidence.md

File metadata and controls

180 lines (137 loc) · 7.43 KB
Raw

CI evidence and the GitHub Action

compare-reports decides whether a run should fail. CI evidence packages why into a reviewable, local bundle — and, inside GitHub Actions, appends a compact summary to the run's step summary. The first-party GitHub Action wires this into a workflow in a few lines.

From regression reports to CI evidence in one GitHub Action.

Everything here is deterministic and local. The library writes files into a directory you name and reads exactly one environment variable (GITHUB_STEP_SUMMARY) when you ask it to append a step summary. No GitHub API calls, no PR comments, no artifact uploads, no GITHUB_TOKEN, no network, no telemetry, no root-cause analysis.

write-ci-evidence vs. compare-reports

Both compare a baseline report with a current one using the same RegressionTriageEngine and the same --fail-on policy and exit codes. The difference is the output:

  • compare-reports optionally writes a JSON and/or Markdown triage report.
  • write-ci-evidence writes a whole evidence directory and can append a GitHub Actions step summary.

compare-reports is unchanged and continues to work exactly as before; write-ci-evidence is additive.

What the bundle contains

write-ci-evidence writes a fixed, sorted set of files into the evidence directory (default entropy-loop-evidence):

  • triage.json — the full triage (counts + ordered transitions).
  • triage.md — the human-readable triage report.
  • summary.txt — a compact plain-text summary.
  • manifest.json — a deterministic manifest (counts, policy, file list).

Only these file names are written, all inside the directory you name — nothing is written or deleted outside it. Output is deterministic (sorted JSON keys, no timestamps, no environment capture), so two runs on the same inputs produce byte-identical files.

Step summary

When --append-github-step-summary is passed (or --github-step-summary PATH), a compact Markdown summary is appended to the target. With the flag and no path, the single environment variable GITHUB_STEP_SUMMARY is used; outside GitHub Actions with no path, nothing is written and the call simply returns without failing. --no-step-summary disables it entirely.

The summary includes the result, the policy, the counts, and the newly failing / fixed / persistent case ids. If the reports carry no per-case data (an older baseline), a compatibility note is shown instead.

The GitHub Action

The repository ships a composite Action at its root (action.yml). It installs entropy-loop-core (optional) and runs write-ci-evidence:

- name: Compare Entropy Loop reports
  uses: koreaelonmusk/entropy-loop-core@v0.8.0
  with:
    baseline-report: baselines/entropy-loop.json
    current-report: reports/current.json
    fail-on: new-failures
    evidence-dir: reports/entropy-loop-evidence
    write-step-summary: true

The Action does not call the GitHub API, comment on PRs, upload artifacts, or require GITHUB_TOKEN. If you want the evidence as an artifact, add actions/upload-artifact yourself — explicitly. See github-actions.md for a full workflow.

Which package version the Action installs

When the Action is pinned to a semver tag such as v0.8.0 and package-version is not set, the Action installs the matching PyPI package version, for example entropy-loop-core==0.8.0.

When the Action is used from a branch ref such as main, and package-version is not set, the Action installs the latest published package from PyPI.

For fully reproducible CI on branch refs, set package-version explicitly.

The version is inferred from the GITHUB_ACTION_REF the runner already provides — no GitHub API call and no GITHUB_TOKEN is involved.

Local CLI equivalent

entropy-loop write-ci-evidence baselines/entropy-loop.json reports/current.json \
  --fail-on new-failures \
  --evidence-dir reports/entropy-loop-evidence \
  --append-github-step-summary

Exit codes: 0 policy passes, 1 policy fails, 2 bad input (missing file, invalid JSON, or invalid policy).

Compatibility

The bundle is built from a v0.7.0 RegressionTriage. Reports generated by v0.6.0 or older may lack case_results, so per-case transitions (and the per-case lines in the step summary) will be empty; the counts and result still render. For accurate per-case evidence, regenerate baselines with v0.7.0 or later. See regression-triage.md.

JUnit XML for CI-native reporting

Many CI systems and test-reporting tools (GitHub Actions, GitLab CI, Jenkins, CircleCI, Buildkite, and others) read JUnit XML. --junit-report writes a deterministic JUnit file so they can consume Entropy Loop regression results as test reports — no custom parsing.

entropy-loop write-ci-evidence baselines/entropy-loop.json reports/current.json \
  --fail-on new-failures \
  --evidence-dir reports/entropy-loop-evidence \
  --junit-report reports/entropy-loop-junit.xml

compare-reports accepts --junit-report too, and it can be combined with --json-report and --markdown-report. The GitHub Action exposes an optional junit-report input that maps to the same flag (empty means no JUnit file).

JUnit failures indicate reported regression/test state; the selected fail-on policy controls the process exit code. So a persistent failure can appear as a JUnit <failure> even when --fail-on new-failures exits 0.

Mapping (transition state only — no root-cause claims):

  • currently-failing cases → <failure> with type new-failure or persistent-failure,
  • resolved and passing cases → passing <testcase>,
  • skipped or missing cases → <skipped> (missing-in-current is skipped, not an error),
  • legacy reports without per-case data → a single regression-summary testcase.

The output uses only the standard library, escapes XML, and contains no timestamps or hostnames, so it is byte-identical across runs on the same input. --junit-report writes an extra file; it does not change the default 4-file evidence bundle.

Human-readable HTML report

For people rather than machines, --html-report writes a self-contained Pixel Failure Console — a single HTML file with inline CSS and no external resources. It combines with the JSON/Markdown/JUnit outputs and, like JUnit, is written outside the default bundle:

entropy-loop write-ci-evidence baselines/entropy-loop.json reports/current.json \
  --fail-on new-failures \
  --evidence-dir reports/entropy-loop-evidence \
  --junit-report reports/entropy-loop-junit.xml \
  --html-report reports/entropy-loop.html

Self-test

The repository dogfoods its own Action: .github/workflows/action-self-test.yml runs the composite Action from the current checkout (uses: ./) on a GitHub-hosted runner, verifies the evidence bundle and step summary, and asserts the write-ci-evidence policy exit codes (0/1/2). A separate job smoke-tests the published v0.8.0 Action as a consumer would use it. The workflow uses least-privilege permissions: contents: read — no GITHUB_TOKEN, no GitHub API, no artifact upload.

Boundary

  • Local files only; explicit paths only.
  • No GitHub API calls, no PR/issue comments, no artifact uploads, no GITHUB_TOKEN, no network.
  • Reads only GITHUB_STEP_SUMMARY, and only when asked.
  • Deterministic output — no timestamps, no environment dump, no secrets.
  • Transition classification only — not root-cause analysis, and no correctness guarantee.