Skip to content

Commit c61e38b

Browse files
ostermanclaude
andauthored
feat: native container steps for workflows and custom commands (#2626)
* feat: native container steps for workflows and custom commands Add a native `type: container` step (build/push/run) to the shared step library used by workflows and custom commands, plus a formal step-outputs contract so one step can build an image and later steps push/run the exact produced artifact. Built on the existing pkg/container Docker/Podman runtime (ephemeral one-shot runner, image build/tag/push/inspect helpers) with per-step identity for registry auth and Docker Buildx Bake support. Includes the examples/container-step example, a hermetic GitHub Actions job that exercises build -> push -> run against a registry:2 service on localhost:5000 (plus failure-propagation), workflow step-type docs, a changelog blog post, and a roadmap update marking container steps and step outputs shipped. Also lands the design PRDs for the follow-on primitives (container components, compose components, and membership-based compositions) and trims the container-step PRD to cover only the procedural step. The earlier targets-based composition scaffolding is removed in favor of those PRDs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix dead core-concepts links in compositions PRD lychee resolves /core-concepts/* as repo-relative file paths, which do not exist; use plain text in the PRD intro instead of website-route links. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(container): parse container ID from last line of docker create output DockerRuntime.Create() returned the full CombinedOutput of `docker create` as the container ID. When the image was missing locally, `docker create` pulled it inline and the pull progress was captured alongside the ID, so the multi-line blob was passed to `docker start`, producing "Error response from daemon: page not found" (the container-step example job). Extract the container ID as the last non-empty line of output via a shared extractContainerID() helper, used by both the Docker and Podman runtimes (de-duplicating Podman's inline loop). Add an empty-ID guard and a regression test covering the exact inline-pull output from CI. Also bundles related container refinements already in the worktree: runtime resolution/recovery ordering in detector.go, error wrapping and cancellation-safe cleanup in ephemeral.go, and PRD doc link fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(container): inject runtime env into container CLI subprocesses Adds an EnvSetter seam so container runtimes (Docker/Podman) run their CLI subprocesses with a materialized environment (e.g. DOCKER_CONFIG for ECR login) instead of always inheriting os.Environ(). Wires the env through the runner container steps (build/push/run) and updates the container-step example, step-type docs, and the native-container-steps blog post. Includes new env_test.go and container_env_test.go coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run linux acceptance tests on the `large` runner to avoid OOM The linux Acceptance Tests job runs `make testacc-cover` with `-coverpkg=./...` across the whole repo, which is memory-heavy. It ran on the `terraform` runner whose RunsOn config has an 8 GiB RAM floor (`ram: [8, 64]`), so spot instances as small as 8 GiB were provisioned. The coverage build was killed (exit 137, "runner received a shutdown signal") at the coverage step on consecutive runs — every test package passed; only the coverage collection OOMed. Switch the linux matrix entry to `runner=large` (`ram: [16, 128]`, `disk: large`), the same runner the `release` job already uses. macos/windows run plain `testacc` (no coverage instrumentation) and are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(container): combine create+pull errors with errors.Join createEphemeralContainer used a single fmt.Errorf with two %w verbs to wrap the create and pull errors. Per the project error-handling guidelines, combine multiple error causes with errors.Join: wrap each cause with its context label and join them under the outer message. Preserves both error chains for errors.Is. Addresses CodeRabbit review on PR #2626. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "ci: run linux acceptance tests on the `large` runner to avoid OOM" This reverts commit b13f9e9. * fix(commands): stop routing shell/atmos custom-command steps through step handlers Custom-command step dispatch had a broad pre-check that sent every *registered* step type (except `exec`) — including `shell` and `atmos` — through the new pkg/runner/step handlers, making the legacy switch cases dead code. That regressed two ways: - Windows: the shell handler hardcodes `exec.Command("sh", "-c", ...)`, so shell-based custom commands (greet/echo/deploy/show) failed with `exit status 126`, breaking their golden snapshots. - Linux: the handler path spawned `atmos` grandchildren without process-group cleanup, leaking orphaned `atmos` processes (32 in one acceptance run vs 0 on main). Accumulated coverage-instrumented `atmos` processes exhausted memory and the on-demand runner was torn down (exit 137 / "runner received a shutdown signal") — which looked like an OOM/spot flake but was a process leak. Gate on stepPkg.IsExtendedStepType instead (matching internal/exec/workflow_utils.go and pkg/workflow/executor.go): shell/atmos/exec keep the legacy, cross-platform, child-reaping paths; only genuinely-extended types (container, input, …) route through the registered handlers via the default case, which now also carries the resolved step env. Removes the now-dead runRegisteredStepHandler helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add container workflow sandbox * fix(workflow): normalize rel path in container workspace-escape guard filepath.Rel returns OS-native separators, so the "../" prefix check in mapHostWorkDirToContainer missed escapes on Windows (..\tmp\outside), letting out-of-workspace working_directory values through. Normalize with filepath.ToSlash before the guard so it is separator-agnostic. Fixes the failing Windows acceptance test TestWorkflowSandboxExecShellRejectsWorkingDirectoryOutsideWorkspace. Also resolve pre-existing golangci-lint findings in the container-sandbox code surfaced by the --new-from-rev gate: - schema: wrap the dynamic container-decode error with a static sentinel (ErrInvalidWorkflowContainer) for err113. - container: pass SandboxConfig by pointer (hugeParam), name the sandbox-name length constant, add perf.Track to Sandbox.ID/Name. - workflow: introduce SandboxParams to keep StartWorkflowSandbox, RunStepContainerOverride and ExecShell within the argument limit; split mergeWorkflowContainer and extract Execute/runCommand helpers to satisfy cyclomatic/function-length; use homedir.Dir over os.UserHomeDir; name the "." constant; convert the shell-dispatch if-else to a switch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(workflow): rename container sandbox file and symbols to container/session The workflow-level container feature is driven by the `container:` block on workflows/steps; "sandbox" leaked in from the lower pkg/container runtime primitive and read as an implementation detail at this layer. Rename the file and the workflow-package symbols to match the feature name. - pkg/workflow/container_sandbox.go -> container.go (+ _test.go) - WorkflowSandbox -> ContainerSession - SandboxParams -> ContainerStepParams - StartWorkflowSandbox -> StartWorkflowContainer - internal helpers/fields: sandboxContainer -> containerBackend, buildSandboxConfig -> buildContainerConfig, ensureWorkflowSandbox / cleanupWorkflowSandbox -> ...Container, ContainerSession.sandbox -> .backend, Executor.sandbox -> .containerSession. The shared pkg/container layer (container.Sandbox, StartSandbox, NewWorkflowSandboxConfig, SandboxConfig, SandboxLabel*, SandboxTypeWorkflow) is left unchanged - it is the correct runtime primitive there. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(container): make workflow sandbox teardown immediate The long-lived workflow sandbox (and step-override ephemeral containers) run `/bin/sh -c "sleep infinity"` as PID 1. The kernel applies no default signal disposition to PID 1, so SIGTERM is ignored — `docker stop`/`podman rm -f` then block for the full 10s stop grace period before sending SIGKILL, adding ~10s to every workflow that uses a container. Set `--stop-signal=SIGKILL` at create time for these keep-alive containers (scoped to OverrideCommand). SIGKILL cannot be caught or ignored, even by PID 1, so stop/rm -f are immediate on both docker and podman. Also drop the now-redundant graceful Stop from Sandbox.Cleanup, aligning it with the ephemeral path which already only force-removes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(container): scope credential mounts, add build spinner, clarify create errors Three fixes to the `type: container` step path, surfaced by running `atmos container build-run` against examples/container-step: 1. credentialFileMounts bind-mounted every absolute-path value in the step env. Because the custom-command path hands the step the entire host environment, it mounted SHELL=/bin/zsh (absent in the runtime VM) and SSH_AUTH_SOCK (a unix socket podman cannot statfs), so `podman create` failed with "statfs … operation not supported". Restrict to an allowlist of credential env keys and only mount existing regular files (skip dirs/sockets/devices). 2. The container build was silent (runtime.Build swallows output on success). Wrap it in spinner.ExecWithSpinner like the devcontainer build path, so the build shows a spinner and a ✓ "Built image <tag>" line (degrades off-TTY). 3. createEphemeralContainer pulled-and-retried on ANY create failure, masking the real cause behind a misleading "pull localhost/<image>" error. Only retry when the create error actually indicates a missing image. Adds the missing unit coverage that let #1 through: credentialFileMounts was untested, and the build-config test fed a curated 2-var env instead of a realistic full host environment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(container): add `inspect` step action and rename example command Add a `type: container, action: inspect` step that renders curated image metadata (ID, digest, tags, created, size, platform, layers, notable OCI labels) as a themed table — not a raw JSON dump. Enrich the runtime ImageInfo and inspect parser with size/created/architecture/os/labels/layers, available from the existing `image inspect --format {{json .}}` output. Also polish the example for newcomers: - Rename the `container` custom command to `example` so it's clear these are user-defined custom commands, not a built-in `atmos container` subcommand. - The `example build-run` flow now builds, inspects (shows metadata), then runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(config): preserve case of nested step-level env keys Viper lowercases all YAML map keys, so a custom-command step's `env:` block (e.g. `EXAMPLE_MESSAGE`) reached the step as `example_message` — a container step's `echo "$EXAMPLE_MESSAGE"` then found nothing. The existing case-sensitive env support (pkg/config/casemap) only collected the top-level `env:` keys. Collect env-var key casing recursively from every `env:` mapping at any depth (top-level, custom-command, and step-level) into the shared "env" case map, and apply it to a custom-command step's declared env before merging it over the command/process env. Command-level `env:` written as a {key,value} list is already case-preserved by Viper and is intentionally left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(container): address PR review feedback CodeRabbit review on #2626: - sandbox: gate the create→pull-retry on image-missing errors only (createSandboxContainer), matching the ephemeral path, so a non-image create failure (e.g. a bad mount) surfaces as-is instead of being masked by a misleading pull error. Join create+pull errors when both fail. - workflow/container: add perf.Track to the public StartWorkflowContainer, ContainerSession.ExecShell, and RunStepContainerOverride entry points; guard params.WorkflowDef/params.Step before dereferencing them; extract resolveStepHostWorkspace to keep RunStepContainerOverride under the cyclomatic limit. - schema: wrap the WorkflowContainer mapping-decode error with ErrInvalidWorkflowContainer so errors.Is works across all input forms. - tests: correct the mismatched doc comment on TestIsImageMissingError and restore it on TestRunEphemeralContainer_PullMissingNonImageErrorDoesNotPull; add createSandboxContainer pull-gating coverage. - docs: document container.shell, container.user, and container.run_args in the workflows parameter reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(container): raise patch coverage on the container-step paths Codecov flagged the PR patch coverage at 59% (below the 80% gate), driven by under-tested new code. Add focused unit tests for the testable logic (no container runtime required): - pkg/workflow/container.go (25.7% -> ~93%): config build + validation, the scalar/collection/toggle merge, host-workdir mapping incl. the workspace-escape guard, mount/port/env conversion, env-slice merge, home expansion, runtime/pull/cleanup validators, dry-run StartWorkflowContainer and RunStepContainerOverride, ExecShell nil guards + dry-run, and the executor's cleanup/ensure/runShellStep container helpers. - pkg/container/sandbox.go (57% -> ~84%): NewWorkflowSandboxConfig, normalizeSandboxConfig, buildSandboxCreateConfig, matchesSandboxLabels, isContainerRunning, sanitizeSandboxName, Sandbox ID/Name/Exec. - pkg/runner/step/output_mode.go (17% -> ~92%): the new ExecuteWithIO family across every mode + error propagation. - pkg/runner/step container handlers: buildSpinnerMessage, validateBuildAction, resolveBuildBake, validateInspectAction, executeInspect dry-run, and the effective-step/resolve helpers. Tests only; no production code changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(container): cover dry-run actions and executor container branches Second coverage pass on the container-step paths, targeting the remaining unit-testable lines Codecov flagged: - step handlers: the dry-run branches of executeRun/executeBuild/executePush (which build their config and render a preview without a runtime), plus validatePushAction, effectivePushStep, metadataString, containerStepName, expandHostPath, resolveRunCommand, and writeOutput. - pkg/workflow/executor.go (51% -> ~90%): the runShellStep step-override and workflow-container branches (both via dry-run / a cached dry-run session) and envSliceToMap. - pkg/container: buildImageInspectArgs. The still-uncovered lines are integration-only: the docker/podman runtime methods that shell out to the real CLI (no injection seam), and the custom-command / workflow execution loops. Those are exercised by the container-step acceptance job, not unit tests. Tests only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(container): end inline test comments with periods (godot) CodeRabbit flagged inline trailing comments missing a terminal period per the repo's godot guideline (godot's default `declarations` scope doesn't catch inline comments, so local lint stayed green). Add the missing periods across the container-step test files. Comments only — no code changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(container): top-level container.runtime namespace + auto-start flag Introduce the top-level `container:` config namespace (forward-compatible with the upcoming container component kind / `atmos container` command) and standardize the docker/podman selector on Atmos's pluggable-system vocabulary, `provider`: ```yaml container: runtime: provider: podman # docker | podman (default: auto-detect) auto_start: true # auto-init/start the Podman machine when none is running ``` - schema: add `ContainerConfig`/`ContainerRuntimeConfig` and the top-level `Container` field; rename the per-step docker/podman selector `runtime:` → `provider:` across ContainerBuild/Push/Run/InspectStep, WorkflowContainer, and WorkflowStep/Task (the buildx `engine:` field is unrelated and unchanged). - config: bridge `container.runtime.auto_start` → the ATMOS_CONTAINER_RUNTIME_AUTO_START env var via PromoteAtmosEnv (env wins over config), so the global default reaches the runtime detector which has no atmosConfig. - container: DetectRuntimeWithPreferenceAndRecovery now ORs in the env flag, so a stopped Podman machine is auto-started for every action and the workflow sandbox without per-step config. Fixes `atmos example build-run` failing with "neither docker nor podman is available" when the machine is stopped. - example: enable `container.runtime.auto_start` so the demo works out of the box. - docs/schema: document the namespace + env override; add the JSON config schema entry; rename the `runtime` step param to `provider`. Adds schema-decode, config-bridge (incl. env-precedence), and detector auto-start (executor-mocked) tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(container): render inspect metadata as a borderless key/value list The inspect output used a Markdown table; glamour padded the "Property" column to a fixed, absurdly wide size. Replace it with a theme-aware, borderless two-column layout: bold keys, no header row, and a column width driven by the longest present key so values align tightly to the content. - renderImageInspect builds the styled string (theme styles, graceful no-color degradation) and the caller prints it via ui.Writeln instead of ui.Markdown. - Extract collectInspectRows so renderImageInspect stays under the cyclomatic limit. - Tests strip ANSI and assert on plain content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(container): restore Markdown heading title for image inspect The flat `Image: <name>` title rendered all-blue (the same Label/Title style as the body keys), so it had no visual hierarchy. Restore the previous Markdown heading (`## Image <name>` → bold heading + the image name as cyan inline code) for the title, while keeping the new borderless bold-key/value body. renderImageInspect now renders only the body; executeInspect prints the heading via ui.Markdown and the body via ui.Writeln. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(container): left-margin the inspect body to align under the heading The key/value body rendered flush-left while the Markdown heading is indented, so they didn't line up. Indent the body block with a lipgloss MarginLeft(2) (idiomatic left margin) so it aligns under the heading. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(container): bottom margin + strip lipgloss padding on inspect table Add a one-line bottom margin under the inspect table to separate it from the following step output, and strip lipgloss's uniform-block right-padding with the shared ANSI-aware per-line trimmer (ansi.TrimLinesRight) so rows carry no trailing whitespace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(container): regenerate describe-config snapshots for container.runtime The new top-level `container:` namespace surfaces a `container.runtime` block in `atmos describe config` output. Regenerate the two affected golden snapshots (via -regenerate-snapshots) so the Acceptance Tests match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: raise container coverage * test: share fake container runtime helper * docs: reorganize workflow docs sidebar * docs(workflows): standardize step Intros, document container nested fields, fix dev-server OOM Website/docs: - Reorder reference sidebar so Workflows sits between Vendor and How-To Guides. - Add the <Intro> component to every workflow step reference page: the parent `type` page, all 27 `type/*` step-type pages, and the remaining workflow-level and step-level field pages. Step-type intros lead with the value: each type exposes an Atmos CLI primitive (interactive prompts, spinner, tables, themed output, command/container execution, structured logging) so users can build their own custom commands and workflows. - Workflow-level `container` field page: example now uses every field, plus dedicated Mounts, Ports, and Per-step overrides sections with concrete nested YAML and accurate merge-vs-replace semantics. - `type: container` step page: per-action Build (+ Bake), Push, Run (+ Mounts/Ports), and Inspect sections with full nested YAML and sub-field definitions, verified against pkg/schema/workflow.go and pkg/workflow. Tooling: - Add website/.npmrc raising Node's heap ceiling (node-options= --max-old-space-size=8192) to stop the Docusaurus dev server and build from running out of memory on this large site. Verified with `pnpm build` (exit 0; no new broken links or anchors). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(commands): split Custom Commands page into per-key subpages The 1,284-line `cli/configuration/commands.mdx` is replaced by a browsable tree mirroring the workflows docs, so each command key has its own reference page and the shared-step-types relationship with workflows is obvious. - New `cli/configuration/commands/` tree (autogenerated under CLI Configuration via `_category_.json`): index (the `commands` array) → `command/` object → per-key pages: arguments, flags, env, steps, component, dependencies, working_directory, identity, and nested commands. Index slug `/cli/configuration/commands` preserved. - The `steps` page leads with "commands run the same step engine as workflows" and links to the shared `/workflows/steps` + `/workflows/steps/type` reference (reusing the `<StepTypes/>` partial) instead of duplicating the 27 step-type pages — confirmed in source: command `Tasks` dispatch through the same `pkg/runner/step` registry as workflow steps. - Repoint all inbound anchor links that moved to subpages, across docs, blog posts, `roadmap.js`, the file-browser plugin (custom-components example sidebar), and two user-facing doc URLs in `cmd/cmd_utils.go`. Verified with `pnpm build` (exit 0) and `go build ./cmd/...`; no new broken links or anchors (the only remaining warning is the pre-existing /changelog page). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(commands): fix two carried-over inaccuracies flagged in review - arguments: drop `required: true` from the `name` example that sets `default: John Doe` (a defaulted argument is optional; the prose already says it falls back when omitted). - identity: separate the temporary credential-file vars (AWS_SHARED_CREDENTIALS_FILE, AWS_CONFIG_FILE) from the profile selector (AWS_PROFILE), which is not a file path. Both were carried over verbatim from the original commands page. Verified with `pnpm build` (exit 0; no new broken links/anchors). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(link-check): exclude no-color.org (unreachable from CI runners) The "Check Markdown Links" job failed with a network connection error to https://no-color.org/ (referenced from docs/prd/help-system-architecture.md and docs/prd/io-handling-strategy.md). The URL is the canonical NO_COLOR spec site and resolves in a real browser; it just intermittently refuses connections from CI runners. Add it to the lychee exclude list, consistent with the existing entries for gnu.org, tldp.org, kubernetes.io, nx.dev, and other valid-but-flaky external links. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 49ce1d3 commit c61e38b

163 files changed

Lines changed: 13677 additions & 2983 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/skills/docs/SKILL.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,17 @@ Every new or changed `atmos.yaml` section needs configuration docs.
3737
- Use `<dl>`, `<dt>`, and `<dd>` for configuration keys and option definitions.
3838
- Include defaults, supported values, and environment variables when they are part of the public interface.
3939

40+
## Sidebar Hierarchy
41+
42+
For configuration docs, make the sidebar resemble the YAML hierarchy.
43+
44+
- Parent categories may link to the page for the object they represent.
45+
- Prefer visible labels that are config keys or object names, such as `workflows`, `workflow`, `steps`, and `env`.
46+
- Avoid editorial labels like "Overview", "Execution", or "Runtime Context" when the page represents a configuration object.
47+
- Do not promote enum values or type-specific parameters to sidebar peers unless they are independent configuration objects.
48+
- When possible, use folder structure plus `_category_.json` so autogenerated sidebar entries inherit the YAML-shaped hierarchy from the docs tree.
49+
- If site-level sidebar sorting prevents YAML-order rendering, use explicit sidebar entries for that section rather than changing global sidebar behavior.
50+
4051
## Command Docs
4152

4253
When command behavior is configured by `atmos.yaml`, link command docs back to configuration docs.

.github/workflows/test.yml

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -765,8 +765,60 @@ jobs:
765765
]
766766
}
767767
768+
# run container step tests against a local registry (build -> push -> run)
769+
# Docker is available directly on the runner (see the k3s job), and a
770+
# registry:2 service container provides a hermetic push/pull target on
771+
# localhost:5000 so the full build -> push -> run cycle is exercised without
772+
# GitHub Container Registry, leftover packages, or fork-PR token limits.
773+
container-step:
774+
name: "[container-step] example"
775+
needs: build
776+
runs-on: ubuntu-latest
777+
778+
services:
779+
registry:
780+
image: registry:2
781+
ports:
782+
- 5000:5000
783+
784+
timeout-minutes: 15
785+
steps:
786+
- name: Download build artifacts
787+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
788+
with:
789+
name: build-artifacts-linux
790+
path: /usr/local/bin
791+
792+
- name: Set execute permissions on atmos
793+
run: chmod +x /usr/local/bin/atmos
794+
795+
- name: Check out code into the Go module directory
796+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
797+
with:
798+
persist-credentials: false
799+
800+
- name: Run container steps (run, build, bake, push, workspace, env)
801+
working-directory: examples/container-step
802+
run: |
803+
set -euo pipefail
804+
atmos workflow hello -f container-step
805+
atmos workflow build-run -f container-step
806+
atmos workflow bake-build-run -f container-step
807+
atmos workflow push-local-registry -f container-step
808+
atmos workflow workspace -f container-step
809+
atmos workflow env -f container-step
810+
811+
- name: Verify a failing container step propagates a non-zero exit code
812+
working-directory: examples/container-step
813+
run: |
814+
if atmos workflow failing-check -f container-step; then
815+
echo "expected the 'failing-check' workflow to fail, but it succeeded"
816+
exit 1
817+
fi
818+
echo "failing-check correctly returned a non-zero exit code"
819+
768820
release:
769-
needs: [test, lint, mock, k3s, floci, floci-go, docker, validate]
821+
needs: [test, lint, mock, k3s, floci, floci-go, docker, validate, container-step]
770822
if: github.event_name == 'push'
771823
uses: cloudposse/.github/.github/workflows/shared-go-auto-release.yml@8244c7c9142e92281e7841f655fa48e9ceb9b454 # main
772824
with:

cmd/cmd_utils.go

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -724,7 +724,7 @@ func executeCustomCommand(
724724
// Determine working directory for command execution.
725725
workDir, err := resolveWorkingDirectory(commandConfig.WorkingDirectory, atmosConfig.BasePath, currentDirPath)
726726
if err != nil {
727-
errUtils.CheckErrorPrintAndExit(err, "Invalid working_directory", "https://atmos.tools/cli/configuration/commands#working-directory")
727+
errUtils.CheckErrorPrintAndExit(err, "Invalid working_directory", "https://atmos.tools/cli/configuration/commands/working-directory")
728728
}
729729
if commandConfig.WorkingDirectory != "" {
730730
log.Debug("Using working directory for custom command", "command", commandConfig.Name, "working_directory", workDir)
@@ -734,7 +734,7 @@ func executeCustomCommand(
734734
// the Atmos process, so it must be the final step and must not set
735735
// supervisor-only fields (tty, interactive, retry, timeout, output).
736736
if err := schema.ValidateExecTasks(commandConfig.Steps); err != nil {
737-
errUtils.CheckErrorPrintAndExit(err, "", "https://atmos.tools/cli/configuration/commands#interactive-and-tty-steps")
737+
errUtils.CheckErrorPrintAndExit(err, "", "https://atmos.tools/cli/configuration/commands/steps#interactive-and-tty-steps")
738738
}
739739

740740
// Initialize step executor once before loop - reused across steps to preserve outputs.
@@ -897,6 +897,14 @@ func executeCustomCommand(
897897
}
898898

899899
// Execute the step based on type.
900+
//
901+
// shell/exec/atmos use the legacy, cross-platform, child-reaping paths
902+
// below; only genuinely-extended step types (container, input, confirm,
903+
// …) route through the registered step handlers via the default case.
904+
// Routing shell/atmos through the handlers regressed Windows (handlers
905+
// hardcode `sh -c` → exit 126) and leaked orphaned `atmos` child
906+
// processes on Linux (no process-group cleanup), so they stay on the
907+
// legacy paths.
900908
switch stepType {
901909
case "shell":
902910
// Execute shell command (backward compatible).
@@ -937,6 +945,20 @@ func executeCustomCommand(
937945
workflowStep := step.ToWorkflowStep()
938946
// Update command with template-resolved value.
939947
workflowStep.Command = commandToRun
948+
// Carry env onto the step so handlers that read step.Env (e.g. the
949+
// container handler's in-container env) see it. The step's own
950+
// declared `env:` had its map keys lowercased by Viper, so restore
951+
// the original case from the shared env case map, then merge it over
952+
// the resolved command/process env (step vars win on collisions).
953+
stepOwnEnv := workflowStep.Env
954+
if atmosConfig.CaseMaps != nil {
955+
stepOwnEnv = atmosConfig.CaseMaps.ApplyCase("env", stepOwnEnv)
956+
}
957+
mergedStepEnv := envSliceToMap(env)
958+
for key, value := range stepOwnEnv {
959+
mergedStepEnv[key] = value
960+
}
961+
workflowStep.Env = mergedStepEnv
940962
// Propagate working directory to extended step if not already set.
941963
if workflowStep.WorkingDirectory == "" {
942964
workflowStep.WorkingDirectory = workDir
@@ -975,6 +997,21 @@ func cloneCommand(orig *schema.Command) (*schema.Command, error) {
975997
return &clone, nil
976998
}
977999

1000+
func envSliceToMap(env []string) map[string]string {
1001+
if len(env) == 0 {
1002+
return nil
1003+
}
1004+
result := make(map[string]string, len(env))
1005+
for _, entry := range env {
1006+
key, value, ok := strings.Cut(entry, "=")
1007+
if !ok {
1008+
continue
1009+
}
1010+
result[key] = value
1011+
}
1012+
return result
1013+
}
1014+
9781015
// findTypedValue finds the value of an argument or flag with the specified semantic type.
9791016
// For arguments, it checks the Type field.
9801017
// For flags, it checks the SemanticType field.

cmd/cmd_utils_test.go

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,22 @@ import (
2424
"github.com/cloudposse/atmos/pkg/schema"
2525
)
2626

27+
func TestEnvSliceToMap(t *testing.T) {
28+
assert.Nil(t, envSliceToMap(nil))
29+
assert.Nil(t, envSliceToMap([]string{}))
30+
assert.Equal(t, map[string]string{
31+
"A": "override",
32+
"B": "2",
33+
"EMPTY": "",
34+
}, envSliceToMap([]string{
35+
"A=1",
36+
"malformed",
37+
"B=2",
38+
"EMPTY=",
39+
"A=override",
40+
}))
41+
}
42+
2743
func TestVerifyInsideGitRepo(t *testing.T) {
2844
// Create a temporary directory for testing
2945
tmpDir := t.TempDir()

docs/prd/compose-components.md

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# PRD: Compose Components
2+
3+
## Summary
4+
5+
Add a `compose` component kind to Atmos. A compose component wraps an existing native
6+
`compose.*.yaml` project (one or more services, e.g. registry + postgres + k3s) as a single Atmos
7+
component. It is the bring-your-own / ecosystem-compatible sibling of the
8+
[container component](container-components.md): Atmos does not reinvent Compose, it runs the project
9+
and layers Atmos config (env/vars interpolation, secrets, identity/auth, and composition membership)
10+
on top.
11+
12+
> A component is one unit. To run a *set* of units together as a system, see
13+
> [compositions](compositions.md).
14+
15+
## Goals
16+
17+
- Operate a native Docker/Podman Compose project as a first-class Atmos component.
18+
- Keep the Compose file 100% native — Atmos never rewrites Compose syntax.
19+
- Layer Atmos config on top: variable interpolation, env, secrets, identity, inheritance/catalogs.
20+
- Provide a deterministic, environment-scoped project name by convention.
21+
- Participate in compositions via the first-class `composition` membership field.
22+
23+
## Non-Goals
24+
25+
- V1 does not surface Docker Compose `profiles` as an Atmos field or flag (see Deferred). Native
26+
`profiles:` inside the user's Compose file still work — Atmos just runs the project.
27+
- V1 does not translate Compose into another backend; a compose component always runs through the
28+
Compose CLI.
29+
- V1 does not model per-service Atmos identity for services inside the Compose file; the component is
30+
the unit of Atmos addressing.
31+
32+
## Public Interface
33+
34+
```yaml
35+
components:
36+
compose:
37+
local-infra:
38+
composition: storefront # first-class membership (see compositions.md)
39+
metadata:
40+
inherits: [compose-defaults] # catalogs / inheritance like any component
41+
files: # native compose file(s), untouched
42+
- compose.local.yaml
43+
project_name: storefront # OPTIONAL; default = sanitized stack name (see below)
44+
env_file:
45+
- .env.local
46+
vars: # interpolated into compose ${...}
47+
POSTGRES_VERSION: "16"
48+
env:
49+
POSTGRES_PASSWORD: !secret pg_password
50+
```
51+
52+
The referenced `compose.local.yaml` stays native Compose:
53+
54+
```yaml
55+
# compose.local.yaml
56+
services:
57+
registry:
58+
image: registry:2
59+
ports: ["5001:5000"]
60+
postgres:
61+
image: "postgres:${POSTGRES_VERSION}"
62+
environment:
63+
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
64+
ports: ["5432:5432"]
65+
k3s:
66+
image: rancher/k3s:latest
67+
privileged: true
68+
ports: ["6443:6443"]
69+
```
70+
71+
## Lifecycle
72+
73+
```bash
74+
atmos compose up local-infra -s local
75+
atmos compose ps local-infra -s local
76+
atmos compose logs local-infra postgres -s local
77+
atmos compose exec local-infra postgres -s local -- psql
78+
atmos compose restart local-infra -s local
79+
atmos compose down local-infra -s local
80+
```
81+
82+
Verb → Compose subcommand mapping:
83+
84+
| atmos verb | compose command |
85+
|--------------------|------------------------------|
86+
| `up` / `deploy` | `compose up -d` |
87+
| `down` / `destroy` | `compose down` (`--volumes`) |
88+
| `ps` | `compose ps` |
89+
| `logs` | `compose logs [service]` |
90+
| `exec` | `compose exec <svc> -- …` |
91+
| `restart` | `compose restart [service]` |
92+
| `build` / `pull` | `compose build` / `pull` |
93+
| `config` | `compose config` (render) |
94+
95+
These verbs match the [composition](compositions.md) fan-out so a compose component operates the same
96+
whether invoked standalone or as a composition member.
97+
98+
## Project Name Convention
99+
100+
`project_name` is a composed convention: derived by default and overridable.
101+
102+
- **Default = the sanitized stack name** (lowercase, `[a-z0-9_-]`, e.g. `plat/ue2-dev` →
103+
`plat-ue2-dev`).
104+
- This makes the Compose project equal to the environment, so `compose ps` / `logs` show the whole
105+
local system for that stack together.
106+
- **Tradeoff**: multiple compose components (or compositions) in one stack share that project
107+
namespace — usually desirable for local dev. Set `project_name` explicitly to separate them.
108+
109+
Container components stay per-component (`atmos-<stack>-container-<name>`) because each is one service;
110+
the compose project groups at the environment level because Compose owns multi-service grouping
111+
natively.
112+
113+
## `container` vs `compose`
114+
115+
| | `container` component | `compose` component |
116+
|---|---|---|
117+
| Unit | One Atmos-native service | A native multi-service Compose project |
118+
| Lifecycle owner | Atmos (labels, named lifecycle) | Compose CLI |
119+
| When to use | Atmos-native per-service control | You already have a `compose.yaml` |
120+
| Grouping | A composition of containers | The Compose project itself |
121+
122+
Both are ordinary stack components with the same component model, membership, and composition fan-out.
123+
124+
## Compose Step (addition to the step library)
125+
126+
Alongside the `compose` component kind, a procedural `type: compose` step **will be added to the shared
127+
step library** — the inline, ephemeral counterpart, mirroring how `type: container` complements the
128+
[container component](container-components.md). It is for operating a Compose project as a step inside a
129+
workflow or custom command (e.g. spin up ephemeral test dependencies, run tests, tear down), without
130+
modeling it as a stack component.
131+
132+
```yaml
133+
steps:
134+
- name: deps-up
135+
type: compose
136+
action: up
137+
files: [compose.test.yaml]
138+
project_name: itest
139+
140+
- name: integration
141+
type: shell
142+
command: go test ./it/...
143+
144+
- name: deps-down
145+
type: compose
146+
action: down
147+
files: [compose.test.yaml]
148+
project_name: itest
149+
```
150+
151+
The step mirrors the component verb mapping (`up`/`down`/`ps`/`logs`/`exec`/`restart`/`build`/`pull`/
152+
`config`) and the same Docker/Podman runtime detection. Because it is a step, it belongs in the shared
153+
step library — its full field reference will be specified in
154+
[Container Actions, Step Outputs, Workflows, and Custom Commands](container-actions-and-step-outputs.md),
155+
next to `type: container`, not in this component PRD. This keeps the procedural step and the declarative
156+
component as separate, complementary surfaces.
157+
158+
| Surface | Shape | Use |
159+
|----------------------|--------------------------------|---------------------------------------|
160+
| `type: compose` step | Procedural, inline, ephemeral | Workflow/custom-command Compose ops |
161+
| `compose` component | Declarative, stack-scoped | Composition member, `atmos compose …` |
162+
163+
## Implementation Notes
164+
165+
- Register `compose` in `schema.Components` and `Components.GetComponentConfig` (sibling of
166+
terraform/helmfile/packer/ansible/container).
167+
- Back the kind with a `pkg/compose` package; re-home the existing `composeArgs()` and
168+
`DetectRuntimeWithPreferenceAndRecovery` from `pkg/composition/service.go`.
169+
- Resolve `vars`/`env` (incl. secrets) into Compose interpolation env and `--env-file`.
170+
- Default `project_name` to the sanitized stack name when unset.
171+
- Resolve the first-class `composition` membership field via normal stack processing.
172+
173+
## Deferred
174+
175+
- **Docker Compose profiles**: not surfaced as an Atmos field or flag in V1. "Profile" conflicts with
176+
the Atmos / Atmos Pro `--profile` (`ATMOS_PROFILE`) config-profile concept. Native compose-file
177+
`profiles:` still work inside the user's Compose file. A general
178+
[labels](compositions.md#deferred) concept is the likely future home for this kind of selection.
179+
180+
## Test Plan
181+
182+
- Schema decode tests for the `compose` component kind and `composition` field.
183+
- Verb → Compose argument construction tests for Docker and Podman.
184+
- `project_name` default sanitization tests (stack name → valid project name).
185+
- Variable / env / secret interpolation into Compose invocation.
186+
- Runtime-gated local smoke test bringing up and tearing down a tiny Compose project.

0 commit comments

Comments
 (0)