freethreaded-danger-probe (ftprobe)

Status: Active development Last updated: 2026-04-28 22:51 EDT

A pure-uv standalone Python project that demonstrates, in three independent probes, the concrete dangers blocking Python 3.14t (free-threaded / no-GIL) deployment for ML pipelines built on PyTorch + Stanza + NumPy + onnxruntime + pyannote.audio.

Designed to be re-run unchanged on Python 3.15 and beyond, so the day a future interpreter line ships we can empirically confirm whether the deployment-readiness criteria have cleared.

The three probes

Probe	What it surfaces	Where it runs
wheels	Wheel-coverage holes in the ML dep set on the active interpreter (both pip and uv pip paths, with divergence reporting).	Anywhere — fast, safe.
concurrency	C-extension thread-safety blockers under N concurrent Python threads sharing one Stanza/PyTorch pipeline.	Long form on disposable hosts; short form anywhere.
soak	The kernel-panic-precursor pattern (long idle + concurrent burst → runaway VM / swap activation / OOM / panic).	Disposable host only. Never CI.

The probe was originally motivated by a prior production kernel-panic incident in our internal infrastructure. The design rationale and the empirical 75-minute soak baseline that did not reproduce that pattern in a controlled VM are documented in our internal workspace; this repo contains the harness, the baselines, and the operator-runnable scripts.

Quick start

# Setup (uv ≥ 0.4)
uv sync --extra ml

# Run a probe
uv run ftprobe wheels      --output results/
uv run ftprobe concurrency --output results/ --duration 15 --threads 4
uv run ftprobe soak        --output results/ --evidence-dir /var/ftprobe/  # disposable host only

Crash-survivable forensic trail (--evidence-dir DIR)

The soak probe is designed for a host that may panic. To extract evidence after a crash, pass --evidence-dir DIR. Per-run output:

DIR/<utc>-<probe>-<host>-<id>/
  ├── RUN_IN_PROGRESS    # sentinel — present iff the run did not finish cleanly
  ├── system_snapshot.txt # OS / hardware / pre-run state
  ├── events.jsonl        # streamed per-sample, fsynced before next sample
  └── result.json         # final ProbeResult — only on clean exit

After a panic-and-reboot, every sample up to the moment of panic is on disk. The presence of RUN_IN_PROGRESS plus the absence of result.json is the signal that the run did not complete; the last line of events.jsonl is the last thing the probe saw before it died.

Re-run procedure when a new Python ships

1. uv python install <new-version> (e.g. 3.15 or 3.15t).
2. uv sync --python <new-version> --extra ml.
3. Run all three probes against it, save JSON.
4. jq / deepdiff against the most recent baseline.
5. Update LINKED-ISSUES.md with cleared blockers.

Output format

Every probe writes a JSON document with a stable schema (see src/ftprobe/_output.py). The schema_version field is bumped only on breaking changes; new fields are added at the end and never renamed, so two runs separated by years are still mechanically diffable.

Tests

uv run pytest

The pytest suite covers the harness logic (output schema, runtime detection, result aggregation, sampler arithmetic) — not the ML libraries themselves. The probes' job is to surface library behavior; the tests' job is to make sure the harness reports it correctly.