Merge pull request #24 from KadenMc/feat/cross-machine-ledger

Cross-machine run ledger

Author: KadenMc

CHANGELOG.md

@@ -7,6 +7,99 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-05-25
+
+### Added
+
+- **Cross-machine run ledger.** New `.aexp/ledger/<job_id>.json` files
+  are sanitized projections of terminal-state runs, committed to git.
+  Every machine sees the same view after `git pull` — the validator
+  no longer needs to read filesystem-local signac workspace state to
+  resolve finding citations. Auto-populated by a hook in
+  `aexp.runs.mark_status` that fires on every terminal-status
+  transition (complete/failed/abandoned/stopped). The projection is an
+  explicit allowlist (statepoint, run_link, status, ended_at,
+  wallclock_s, tracker pointers, code commit, registered_machine,
+  promoted_at). Per-machine debris (absolute paths in
+  `tracker_log/events.jsonl`, wandb offline-run dirs, user artifacts)
+  stays in the gitignored `.runs/workspace/<id>/`. New `aexp.ledger`
+  module with `promote_to_ledger`, `backfill_ledger`, and lookup
+  helpers.
+- **`aexp ledger promote <id>` / `aexp ledger backfill` CLI verbs.**
+  Manual one-shot promotion and a bulk migration tool. Each machine
+  with terminal-state runs runs `aexp ledger backfill` once after
+  upgrading; the resulting ledger files commit and push.
+- **`aexp validate --strict-runs={error|warn|off}` flag.** Manual
+  severity knob for finding-citation existence checks. Default
+  `error` preserves 0.5 behavior. `warn` downgrades existence
+  failures to warnings (exit 0); `off` skips existence checks
+  entirely. Structural-shape checks always emit at error severity
+  regardless. Useful while the ledger is being backfilled across
+  machines.
+- **`finding.absent_run_citation` warning code** for citations that
+  resolve in a per-machine index file but not in the local store or
+  ledger — distinguishes "lives on another machine" from "broken."
+  Same treatment for batches: `finding.absent_batch_runs`.
+- **`finding.no_run_store` warning code** emitted once per validate
+  run when no source-of-truth is available (no ledger, no local
+  store, no index). Replaces a previously-silent tolerance branch
+  that made it easy to overlook a missing local store.
+- **`.aexp/installed.json::machine_label` field.** Short identifier
+  for each install, tagged on ledger entries via `registered_machine`.
+  Default: short hostname. Override at install time with
+  `aexp install --machine-label <name>`, or by editing
+  `installed.json` directly. Sticky across re-installs. New
+  `read_machine_label` helper in `aexp.utils.paths`.
+- **`aexp install` manages a `.gitignore` block.** Block-merged with
+  `# agentic-experiments:begin`/`:end` markers. Body sets `.aexp/*` +
+  `!.aexp/runs-index/` + `!.aexp/ledger/` so per-machine state stays
+  ignored while cross-machine shared subdirs are committable. Emits
+  a `gitignore_migration_warning` action when a legacy `.aexp/`
+  pattern is detected outside the managed block (the legacy form
+  parent-excludes `.aexp/`, blocking the `!` exceptions).
+- **`aexp runs-export-index` verb (transitional).** Per-machine JSON
+  dump of terminal runs at `.aexp/runs-index/<machine_label>.json`,
+  unioned by the validator into a three-state vocabulary. Deprecated
+  immediately in favor of `aexp ledger backfill`; emits a
+  deprecation warning when invoked. Kept one release window for
+  back-compat reads.
+
+### Changed
+
+- **`aexp.runs.mark_status` is the unified terminal-status hook
+  point.** `run_lifecycle`'s `complete`/`failed` status writes route
+  through `mark_status` (with `set_ended_at=False` to avoid an extra
+  doc-load inside the Windows file-lock race window during
+  `stop_queued` shutdown). The function gains a `set_ended_at: bool
+  = True` keyword; default preserves the pre-0.6 behavior for
+  existing callers. New `TERMINAL_STATUSES` module constant.
+- **`aexp.linking.link_to_experiment` re-promotes on terminal jobs.**
+  Re-stamping a job's run-link via `aexp link` now also updates the
+  ledger entry's `run_link` field, so the ledger projection doesn't
+  lag the on-disk state.
+
+### Fixed
+
+- Validator no longer silently passes finding citations when neither
+  a local run store nor any cross-machine source exists — surfaces a
+  `finding.no_run_store` warning so the silence is visible.
+
+### Migration
+
+Existing consumers upgrading from 0.5:
+
+1. `pip install -e <agentic-experiments>` in each env.
+2. `aexp install --force` once per consumer (writes the new
+   gitignore block, materializes `machine_label`). If a
+   `gitignore_migration_warning` is emitted, delete the legacy
+   `.aexp/` line from `.gitignore`.
+3. `aexp ledger backfill` on each machine that has terminal-state
+   runs.
+4. Commit `.gitignore` and `.aexp/ledger/`. Push, then pull on every
+   other consumer.
+
+See `docs/queue.md` § cross-machine for the full workflow.
+
 ## [0.5.0] - 2026-05-21
 
 ### Changed

docs/queue.md

@@ -494,6 +494,107 @@ recommended pattern is:
 - Cluster writes **during/after** execution (run-queued → status transitions).
 - Laptop doesn't touch `.runs/` while the cluster is running.
 
+### Validator and citations across machines
+
+If a `kb/research/findings/F###.md` cites a job that lives on the
+cluster but not your laptop, the validator on the laptop can't see
+the run directly — the signac project listing is filesystem-local.
+Before 0.6 this was a hard error. As of 0.6 the validator has three
+layered sources of truth, in order of preference:
+
+1. **`.aexp/ledger/<job_id>.json`** — the universal cross-machine
+   ledger. Sanitized projection of a terminal-state job, committed
+   to git. Every machine sees the same ledger after `git pull`. This
+   is the canonical "this run exists" source.
+2. **Local `.runs/workspace/<id>/`** — the signac project. Counts as
+   "here" too, so freshly-promoted runs (after `mark_status` fires
+   the auto-promote hook but before the next git push) are still
+   recognized.
+3. **`.aexp/runs-index/<machine>.json`** — a transitional per-machine
+   index. Deprecated; kept one release for back-compat. A citation
+   matching only an index file (not in the ledger or local store)
+   emits `finding.absent_run_citation` (warning) so you know a peer
+   machine has the run but it hasn't been promoted yet.
+
+The steady-state workflow:
+
+```
+# cluster (after each batch finishes):
+aexp ledger backfill          # promotes every terminal-state local run
+git add .aexp/ledger/
+git commit -m "ledger: backfill <date>"
+git push
+
+# laptop:
+git pull
+aexp validate                  # exits 0 — every cited job resolves via ledger
+```
+
+The auto-promote hook in `aexp.runs.mark_status` makes the backfill
+step optional in steady state — every `mark_status(job, "complete")`
+already writes the ledger entry. `aexp ledger backfill` is the
+migration path for runs registered before 0.6 (or in environments
+where the hook didn't fire for some reason).
+
+#### --strict-runs escape hatch
+
+If the ledger hasn't been backfilled everywhere yet (or you're
+running against an older install), `aexp validate
+--strict-runs={error|warn|off}` lets you downgrade or skip the
+existence check:
+
+```
+aexp validate                       # exits 1: "ERROR finding.broken_run_citation ..."
+aexp validate --strict-runs=warn    # exits 0: "WARNING finding.broken_run_citation ..."
+aexp validate --strict-runs=off     # skips existence checks entirely
+```
+
+`warn` is the right escape hatch during a partial rollout.
+Structural-shape checks (malformed citation, non-32-hex ids) always
+emit at error severity regardless — those are real authoring
+mistakes, not cross-machine ledger gaps.
+
+### Migrating to the cross-machine ledger (0.5 → 0.6)
+
+After upgrading aexp:
+
+1. On every machine that has terminal-state runs:
+   ```
+   pip install -e <agentic-experiments>     # (you do this)
+   aexp install --force                      # picks up new gitignore block + scaffold
+   aexp ledger backfill                      # promotes every terminal-state job
+   git add .gitignore .aexp/ledger/
+   git commit -m "migrate to aexp 0.6 ledger"
+   git push
+   ```
+2. On the laptop (or any other consumer of these runs):
+   ```
+   git pull
+   aexp validate    # exits 0 — every cited job resolves via ledger
+   ```
+3. The transitional `aexp runs-export-index` verb still works for
+   one release window, with a deprecation warning. Remove
+   `.aexp/runs-index/` after every machine has backfilled.
+
+If `aexp install --force` warns about a legacy `.aexp/` pattern in
+your gitignore (`gitignore_migration_warning`), edit `.gitignore` and
+remove the plain `.aexp/` line — git can't re-include
+`.aexp/runs-index/` or `.aexp/ledger/` while the whole `.aexp/`
+directory is excluded. The aexp-managed block (with `.aexp/*` +
+explicit `!` exceptions) replaces it.
+
+### Machine identity
+
+The ledger entries record which machine registered each run via the
+`registered_machine` field, sourced from
+`.aexp/installed.json::machine_label`. Default is short hostname
+(`socket.gethostname().split(".")[0]`); override at install time with
+`aexp install --machine-label <name>`, or by editing
+`.aexp/installed.json` directly (the file is per-machine and
+gitignored). On HPC clusters where per-node hostnames are noisy,
+set `--machine-label cluster` so the ledger is consistent across
+login nodes and compute nodes.
+
 ## FAQ
 
 **What happens if the runner script is killed mid-job?**

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agentic-experiments"
-version = "0.5.0"
+version = "0.6.0"
 description = "Git-first, hypothesis-forcing experiment tracking for agent-driven ML research. Bundles a research harness for the H->E->F artifact model, uses signac for local execution/run state, and bridges to W&B for remote observability."
 authors = [
     {name = "Kaden McKeen", email = "mckeenkaden@gmail.com"}