Compatibility-matrix test gate for experimental → stable feature graduation

[MaxymVlasov]

Describe the Feature

An automated feature-compatibility test matrix that runs as a required CI gate whenever a command's IsExperimental() flips to false — before a feature loses its [EXPERIMENTAL] badge and graduates to stable, CI automatically tests it in combination with a registry of already-stable feature dimensions (vendoring, remote imports, multi-file stacks, custom commands, templating functions, backends, workdir modes, etc.), using pairwise (all-pairs) generation to keep the test count manageable instead of a full combinatorial explosion.

Expected Behavior

When a PR flips a command's IsExperimental() to false, CI should auto-generate the pairwise matrix for that feature against the current stable-dimension registry and run the feature's test suite against each generated combination — failing the PR if any combination breaks. Today, nothing like this runs: a feature can go from experimental to stable with no more testing than it already had in isolation.

Use Case

Isn't it weird to find out that a new stable feature doesn't work with all the old stable features? Say a command ships under experimental — it works in a pretty janky, barely-there way for about 1.5 use cases. Fine, that's exactly what experimental is for. But then when it graduates to stable, shouldn't there be some kind of check confirming it actually works together with all the already-known, existing features? Right now there isn't.

Atmos' surface area is big and features interact constantly — vendoring + remote imports, workdir + backend provisioning, toolchain + CI, custom commands + stack imports, etc. A feature that looks fine in isolation during its experimental phase (especially if it's only really been stress-tested for ~1-2 use cases by early adopters who click past the [EXPERIMENTAL] warning) can still break once combined with everything else people already rely on in stable. That combination is exactly the class of bug that tends to surface only after something ships as stable.

The experimental marking system itself is genuinely solid — settings.experimental (silence/disable/warn/error), ATMOS_EXPERIMENTAL, [EXPERIMENTAL] badges (#1940). But the graduation side stops at "we should have one someday" — docs/prd/experimental-features-system.md lists a 3-stage path (Experimental → Beta → Stable) explicitly under Future Considerations (not implemented), and its own "Testing" section only covers the flag mechanism itself, not cross-feature compatibility. The tests/ suite isn't thin either (solid CLI/emulator/snapshot tests, e2e workflows like validation-e2e.yml/native-ci.yml) — it's just organized per-feature, with nothing gating combinations at the graduation moment specifically.

Describe Ideal Solution

1. Registry of stable dimensions — a fixture/config catalog of stable features/configs likely to interact with a graduating feature (vendoring on/off, remote imports, multi-file/parent-scoped stacks, custom commands, templating/YAML functions, multiple backends, workdir modes, toolchain-managed vs system tools, etc.). Raw material already exists: tests/fixtures, plus emulator_cloud_test.go / emulator_kubernetes_test.go / emulator_native_aws_test.go / emulator_openbao_test.go / emulator_persistence_test.go / emulator_registry_test.go.
2. Pairwise (all-pairs) generation instead of true N×N — covers every pair of dimensions with a roughly-linear case count instead of exponential (the standard PICT/allpairs technique for exactly this problem).
3. Gate tied to IsExperimental() — flipping it to false in a PR auto-triggers the matrix; a broken combination blocks the merge.
4. Phased rollout: (1) build registry + generator, run informationally against the 5 current experimental commands to baseline; (2) make it blocking specifically for IsExperimental() → false PRs; (3) keep the registry living as new stable features ship.

Alternatives Considered

• Documented manual checklist, no automation — too easy to skip/forget, doesn't scale with maintainer time.
• Full N×N combinatorial matrix — correct in theory but explodes fast; pairwise gets most of the bug-catching value for a fraction of the test count.
• Status quo (no formal check, rely on user bug reports post-stable) — works today but graduation quality stays unverified until real users hit it in production.

Additional Context

• docs/prd/experimental-features-system.md — see "Graduation Path" (under Future Considerations) and "Testing"
• feat: Add experimental feature transparency and controls #1940 — introduced settings.experimental and the [EXPERIMENTAL] badge system
• https://atmos.tools/changelog/experimental-feature-flag — current experimental feature list
• Open to LLM-assisted test generation helping bootstrap/maintain the dimension registry and pairwise cases, if useful at this project's scale — happy to discuss either way.