Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 35 additions & 8 deletions .claude/skills/field-test/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: field-test
description: "Hands-on manual DX test pass of a feature or CLI command: read the real implementation and tests, hypothesize plausible user misunderstandings and misuse automated tests don't cover, build durable fixtures, execute for real against real state, and report ranked findings. Investigation only — never fixes anything found. Invoke on explicit requests like 'field test X' / 'do a DX test pass on X' / 'find vibe-coded slop in X'."
argument-hint: "Feature or command to test, e.g. 'atmos vendor pull'"
description: "Hands-on manual DX test pass of a feature or CLI command: read the real implementation and tests, hypothesize plausible user misunderstandings and misuse automated tests don't cover, build durable fixtures, execute for real against real state, and report ranked findings. Investigation only — never fixes anything found. Defaults to testing whatever the current branch changed vs its base branch when no explicit target is given. Invoke on explicit requests like 'field test X' / 'do a DX test pass on X' / 'find vibe-coded slop in X' / 'field test this branch'."
argument-hint: "Feature or command to test, e.g. 'atmos vendor pull' (omit to default to this branch's change)"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
metadata:
copyright: Copyright Cloud Posse, LLC 2026
version: "1.0.0"
Expand All @@ -10,8 +10,24 @@ metadata:
# Field Test

Hands-on, adversarial test pass of **`$ARGUMENTS`** (the feature/command named when this skill
was invoked, e.g. `atmos vendor pull`). If no target was given, ask which feature/command to test
before starting.
was invoked, e.g. `atmos vendor pull`).

**If no target was given, default to the change introduced on the current branch** rather than
asking. Resolve the actual pull-request base branch when one is available
(`gh pr view --json baseRefName -q .baseRefName` for the current branch), falling back to the
repository's default branch (`gh repo view --json defaultBranchRef -q .defaultBranchRef.name`, or
`origin/main`/`main` if `gh` isn't available) when no PR exists yet. Do NOT use the upstream
tracking branch (`@{u}`) as the base — for a normal feature branch that tracks
`origin/<same-branch-name>`, diffing against its own upstream produces an empty or near-empty
diff, not the PR's actual changes, once the branch has been pushed. Then inspect the FULL set of
changes relative to that base: `git diff <base>...HEAD --stat` for committed history, plus
`git status --porcelain` and `git diff HEAD` for any staged, unstaged, or untracked changes not
yet committed — a field test run before the day's work is committed must still see it. Derive the
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
test target from all of that — the CLI command(s), flag(s), config option(s), or subsystem the
changed files implement — and state explicitly what you inferred and why before proceeding to
Phase 1. Only fall back to asking the user if there's truly nothing changed (clean worktree, base
equals HEAD) or the changes span multiple unrelated features with no coherent single target (ask
which one to focus on, don't silently pick one).

Goal: catch "vibe-coded slop" — behavior that looks fine in code review but breaks or misleads a
real user — not to re-run what automated tests already cover. Anticipate plausible user
Expand All @@ -27,10 +43,21 @@ The goal is a map of "documented or plausible usage" minus "already tested" = wh
verification. This phase is broad, read-only research — delegate it to `Agent subagent_type:
"Explore"` (1-3 agents in parallel, one per bullet below) rather than doing it all serially inline.

- **Implementation** — the actual code, not just its docs or the skill describing it. Per this
repo's conventions, business logic lives in narrow `pkg/` packages, not `internal/exec/` (being
phased out) — check both `cmd/<command>/` (thin call site) and the `pkg/` package(s) it
delegates to for the real logic and error paths.
When defaulting to the current branch (no explicit target given), scope every bullet below to the
target inferred from the branch diff — don't research the whole surrounding subsystem when the
branch only touched one corner of it. If the branch's changed files span more than one command or
package, treat each as a separate target to cover in Phase 2-4, prioritized by how much of the
diff each accounts for.

- **Implementation** — the actual code, not just its docs or the skill describing it. Inspect
every changed production package identified by the diff — new business logic belongs in narrow
`pkg/` packages per this repo's conventions, but `internal/exec/` is still where a large amount
of existing logic lives during its ongoing migration, so a branch touching files there must
still be read, not skipped. Check both `cmd/<command>/` (thin call site) and whichever
`pkg/`/`internal/exec/` package(s) it delegates to for the real logic and error paths. When
defaulting from a branch diff, read the diff itself first (not just the post-change files) —
the diff shows what changed *from*, which is where a regression or half-finished edge case
would show up.
- **Docs and skills** — every relevant page under `website/docs/cli/commands/`, the matching
`.claude/skills/atmos-*` skill(s) for the subsystem, and any README describing the feature. Note
anything phrased with confidence you haven't independently confirmed against the code — docs and
Expand Down
42 changes: 32 additions & 10 deletions cmd/config/operations.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,46 @@ import (
cfg "github.com/cloudposse/atmos/pkg/config"
"github.com/cloudposse/atmos/pkg/data"
"github.com/cloudposse/atmos/pkg/perf"
"github.com/cloudposse/atmos/pkg/schema"
"github.com/cloudposse/atmos/pkg/ui"
u "github.com/cloudposse/atmos/pkg/utils"
atmosyaml "github.com/cloudposse/atmos/pkg/yaml"
)

// valueType holds the --type flag for `config set`.
var valueType string

var configGetCmd = &cobra.Command{
Use: "get <path>",
Short: "Read a value from atmos.yaml by dot-notation path",
Long: "Read a value from atmos.yaml using a dot-notation path (e.g. logs.level).",
Use: "get <path>",
Short: "Read a value from the effective Atmos configuration by dot-notation path",
Long: `Read a value using a dot-notation path (e.g. logs.level) from the effective,
fully-merged Atmos configuration for this invocation -- the same configuration
"terraform plan", "list stacks", etc. actually use, including every --config
file, --config-path directory, and profile applied on top of each other. This
can differ from what a single physical atmos.yaml file declares on its own
(cloudposse/atmos#2867): use "atmos config format" or read the file directly
to inspect one file's own declared value instead.`,
Example: "atmos config get logs.level",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
defer perf.Track(atmosConfigPtr, "config.getRunE")()

file, err := resolveConfigFile(cmd)
// Reload rather than reuse atmosConfigPtr, mirroring configListCmd: this keeps `get`
// independently correct (and independently testable via RunE) even before root.go's
// PersistentPreRun has populated the package-level pointer. Safe to call with an empty
// ConfigAndStacksInfo{} -- LoadConfig now falls back to os.Args/env for
// --config/--config-path/--base-path itself (cloudposse/atmos#2868).
atmosConfig, err := cfg.InitCliConfig(schema.ConfigAndStacksInfo{}, false)
if err != nil {
return err
}

effectiveYAML, err := u.ConvertToYAML(atmosConfig)
if err != nil {
return err
}

value, err := atmosyaml.GetFile(file, args[0])
value, err := atmosyaml.Get([]byte(effectiveYAML), args[0])
if err != nil {
return err
}
Expand Down Expand Up @@ -140,12 +158,16 @@ func init() {
}

// resolveConfigFile picks the atmos.yaml to edit. The inherited persistent
// --config flag (first entry) acts as an explicit override; otherwise the file
// is discovered in the current directory or git root.
// --config flag acts as an explicit override when it names exactly one file;
// otherwise the file is discovered in the current directory or git root.
func resolveConfigFile(cmd *cobra.Command) (string, error) {
override := ""
if cfgFiles, _ := cmd.Flags().GetStringSlice("config"); len(cfgFiles) > 0 {
override = cfgFiles[0]
cfgFiles, _ := cmd.Flags().GetStringSlice("config")
override, err := cfg.ResolveConfigOverride(cfgFiles)
if err != nil {
return "", errUtils.Build(errUtils.ErrInvalidArgumentError).
WithExplanation(err.Error()).
WithHint("Pass a single --config file to config set/delete/format, or edit the target file directly.").
Err()
}

file, err := cfg.ResolveEditableConfigFile(atmosConfigPtr, override)
Expand Down
110 changes: 110 additions & 0 deletions cmd/config/operations_test.go
Original file line number Diff line number Diff line change
@@ -1,15 +1,22 @@
package config

import (
"bytes"
stdio "io"
"os"
"path/filepath"
"strings"
"testing"

ckerrors "github.com/cockroachdb/errors"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

errUtils "github.com/cloudposse/atmos/errors"
"github.com/cloudposse/atmos/pkg/data"
iolib "github.com/cloudposse/atmos/pkg/io"
"github.com/cloudposse/atmos/pkg/schema"
atmosyaml "github.com/cloudposse/atmos/pkg/yaml"
)
Expand Down Expand Up @@ -130,6 +137,29 @@ func TestResolveConfigFile_Error(t *testing.T) {
require.ErrorIs(t, err, errUtils.ErrInvalidArgumentError)
}

// TestResolveConfigFile_MultipleConfigFilesAmbiguous guards against a bug found during a
// field-test pass on cloudposse/atmos#2867/#2868: resolveConfigFile silently used only the
// FIRST --config file (cfgFiles[0]) when multiple were given, so `config set --config a,b
// logs.level X` reported success editing a.yaml while the actual effective value (what `config
// get` reports, and what every other atmos command uses) stayed unchanged whenever b.yaml also
// set that key -- a false success, not just a stale-value bug.
func TestResolveConfigFile_MultipleConfigFilesAmbiguous(t *testing.T) {
dir := t.TempDir()
fileA := filepath.Join(dir, "a.yaml")
fileB := filepath.Join(dir, "b.yaml")
require.NoError(t, os.WriteFile(fileA, []byte("settings:\n enabled: true\n"), 0o644))
require.NoError(t, os.WriteFile(fileB, []byte("settings:\n enabled: false\n"), 0o644))

cmd := &cobra.Command{}
cmd.Flags().StringSlice("config", []string{fileA, fileB}, "")

_, err := resolveConfigFile(cmd)
require.ErrorIs(t, err, errUtils.ErrInvalidArgumentError)
details := strings.Join(ckerrors.GetAllDetails(err), "\n")
assert.Contains(t, details, "a.yaml")
assert.Contains(t, details, "b.yaml")
}

func TestConfigGetCommand_MissingValue(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "atmos.yaml")
Expand All @@ -146,6 +176,86 @@ func TestConfigGetCommand_MissingValue(t *testing.T) {
require.ErrorIs(t, err, atmosyaml.ErrYAMLPathNotFound)
}

// configGetTestStreams is a minimal io.Streams implementation for capturing data output,
// mirroring configSchemaTestStreams in schema_test.go.
type configGetTestStreams struct {
stdin stdio.Reader
stdout *bytes.Buffer
stderr *bytes.Buffer
}

func (ts *configGetTestStreams) Input() stdio.Reader { return ts.stdin }
func (ts *configGetTestStreams) Output() stdio.Writer { return ts.stdout }
func (ts *configGetTestStreams) Error() stdio.Writer { return ts.stderr }
func (ts *configGetTestStreams) RawOutput() stdio.Writer { return ts.stdout }
func (ts *configGetTestStreams) RawError() stdio.Writer { return ts.stderr }

// TestConfigGetCommand_ReportsEffectiveMergedValue reproduces the "stale value" half of
// cloudposse/atmos#2867: `atmos config get` used to read only the FIRST --config file
// directly off disk (resolveConfigFile picked cfgFiles[0]), so a second --config file's
// override was invisible to `get` even though the rest of atmos (stack discovery, etc.)
// correctly used the merged value. `get` must report the same effective value everything
// else uses.
func TestConfigGetCommand_ReportsEffectiveMergedValue(t *testing.T) {
dir := t.TempDir()
mainFile := filepath.Join(dir, "main.yaml")
fragmentFile := filepath.Join(dir, "fragment.yaml")

require.NoError(t, os.WriteFile(mainFile, []byte(`
base_path: "."
stacks:
base_path: "stacks"
included_paths:
- "deploy/**/*"
`), 0o644))
require.NoError(t, os.WriteFile(fragmentFile, []byte(`
stacks:
included_paths:
- "deploy/**/*"
- "other/**/*"
`), 0o644))

streams := &configGetTestStreams{stdin: &bytes.Buffer{}, stdout: &bytes.Buffer{}, stderr: &bytes.Buffer{}}
ioCtx, err := iolib.NewContext(iolib.WithStreams(streams))
require.NoError(t, err)
data.InitWriter(ioCtx)
t.Cleanup(data.Reset)

viper.Reset()
t.Cleanup(viper.Reset)

origArgs := os.Args
t.Cleanup(func() { os.Args = origArgs })
os.Args = []string{"atmos", "--config", mainFile + "," + fragmentFile, "config", "get", "stacks.included_paths"}

require.NoError(t, configGetCmd.RunE(configGetCmd, []string{"stacks.included_paths"}))
Comment thread
coderabbitai[bot] marked this conversation as resolved.

output := streams.stdout.String()
assert.True(t, strings.Contains(output, "deploy/**/*"), "output should contain the first file's value: %s", output)
assert.True(t, strings.Contains(output, "other/**/*"),
"output must reflect the SECOND --config file's override, not just the first file's stale value: %s", output)
}

// TestConfigGetCommand_InitCliConfigError proves configGetCmd.RunE surfaces a genuine
// InitCliConfig failure (a malformed --config file here) instead of panicking or masking it,
// since `get` now reloads the full effective config on every invocation rather than reading a
// single already-validated file (cloudposse/atmos#2867/#2868).
func TestConfigGetCommand_InitCliConfigError(t *testing.T) {
dir := t.TempDir()
badFile := filepath.Join(dir, "bad.yaml")
require.NoError(t, os.WriteFile(badFile, []byte("settings:\n enabled: [true\n"), 0o644)) // unterminated flow sequence

viper.Reset()
t.Cleanup(viper.Reset)

origArgs := os.Args
t.Cleanup(func() { os.Args = origArgs })
os.Args = []string{"atmos", "--config", badFile, "config", "get", "settings.enabled"}

err := configGetCmd.RunE(configGetCmd, []string{"settings.enabled"})
require.Error(t, err)
}
Comment on lines +243 to +257

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Isolate command state with cmd.NewTestKit(t).

Create cmd.NewTestKit(t) before invoking configGetCmd.RunE. This test executes a Cobra command and can otherwise retain shared command state across tests.

As per coding guidelines: “Always use cmd.NewTestKit(t) in command tests to isolate and clean root-command state.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/config/operations_test.go` around lines 243 - 257, Update
TestConfigGetCommand_InitCliConfigError to create and use cmd.NewTestKit(t)
before invoking configGetCmd.RunE, ensuring the Cobra command test isolates and
cleans shared root-command state while preserving the existing malformed-config
assertions.

Source: Coding guidelines


func TestConfigSetCommand_TypeVariants(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "atmos.yaml")
Expand Down
37 changes: 36 additions & 1 deletion internal/exec/describe_workflows_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,11 @@ workflows:
// Update config with the correct base path
config.BasePath = tmpDir
config.Workflows.BasePath = "stacks/workflows"
// Refresh derived absolute paths (e.g. WorkflowsDirAbsolutePath) after mutating BasePath
// directly -- real callers only ever get a fresh AtmosConfiguration from InitCliConfig,
// which computes these together; a manual post-load field mutation must recompute them
// too or ExecuteDescribeWorkflows would resolve against the stale pre-mutation paths.
require.NoError(t, cfg.AtmosConfigAbsolutePaths(&config))

tests := []struct {
name string
Expand Down Expand Up @@ -120,7 +125,7 @@ workflows:
},
},
wantErr: true,
errContains: "the workflow directory 'nonexistent' does not exist",
errContains: "workflow directory does not exist: 'nonexistent'",
},
}

Expand Down Expand Up @@ -190,6 +195,7 @@ workflows:
config := initTestConfig(t)
config.BasePath = tmpDir
config.Workflows.BasePath = "stacks/workflows"
require.NoError(t, cfg.AtmosConfigAbsolutePaths(&config))

tests := []struct {
name string
Expand Down Expand Up @@ -287,6 +293,7 @@ workflows:
config := initTestConfig(t)
config.BasePath = tmpDir
config.Workflows.BasePath = "stacks/workflows"
require.NoError(t, cfg.AtmosConfigAbsolutePaths(&config))

// Should continue processing and return valid workflows despite invalid file.
listResult, _, _, err := ExecuteDescribeWorkflows(config)
Expand Down Expand Up @@ -326,6 +333,7 @@ some_other_key:
config := initTestConfig(t)
config.BasePath = tmpDir
config.Workflows.BasePath = "stacks/workflows"
require.NoError(t, cfg.AtmosConfigAbsolutePaths(&config))

// Should continue processing and return valid workflows.
listResult, _, _, err := ExecuteDescribeWorkflows(config)
Expand All @@ -344,6 +352,7 @@ func TestExecuteDescribeWorkflows_EmptyWorkflowsDirectory(t *testing.T) {
config := initTestConfig(t)
config.BasePath = tmpDir
config.Workflows.BasePath = "stacks/workflows"
require.NoError(t, cfg.AtmosConfigAbsolutePaths(&config))

// Empty workflows directory.
listResult, mapResult, allResult, err := ExecuteDescribeWorkflows(config)
Expand All @@ -353,3 +362,29 @@ func TestExecuteDescribeWorkflows_EmptyWorkflowsDirectory(t *testing.T) {
assert.Len(t, mapResult, 0)
assert.Len(t, allResult, 0)
}

// TestExecuteDescribeWorkflows_PathLeak guards against a bug found during a field-test pass on
// cloudposse/atmos#2867/#2868: "the workflow directory '%s' does not exist" interpolated the
// resolved absolute workflowsDir directly, and the sibling "error reading the directory" message
// interpolated the raw, unresolved atmosConfig.Workflows.BasePath instead of workflowsDir (the
// directory actually searched) -- a copy/paste inconsistency with the sibling branch one line up.
// The pre-existing "nonexistent workflows directory" test above doesn't catch either bug: it
// leaves BasePath/WorkflowsDirAbsolutePath unset, so getWorkflowsDirToUse falls back to a bare
// relative path that was never absolute to begin with.
func TestExecuteDescribeWorkflows_PathLeak(t *testing.T) {
resolvedDir := resolvedTempDir(t)

config := schema.AtmosConfiguration{
BasePath: resolvedDir,
Workflows: schema.Workflows{
BasePath: "nonexistent-workflows-dir",
},
}
require.NoError(t, cfg.AtmosConfigAbsolutePaths(&config))

_, _, _, err := ExecuteDescribeWorkflows(config)

require.Error(t, err)
assert.NotContains(t, err.Error(), resolvedDir, "error must not leak the machine-specific absolute directory")
assert.Contains(t, err.Error(), "nonexistent-workflows-dir")
}
14 changes: 14 additions & 0 deletions internal/exec/validate_component.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ func getBasePathToUse(atmosConfig *schema.AtmosConfiguration) string {
return atmosConfig.BasePath
}

// getWorkflowsDirToUse returns the appropriate workflows directory for file resolution. It
// prefers the precomputed WorkflowsDirAbsolutePath (set by AtmosConfigAbsolutePaths, the same
// mechanism cloudposse/atmos#2864 uses for the top-level base_path), falling back to joining
// the raw BasePath/Workflows.BasePath for callers that construct an AtmosConfiguration by hand
// without running it through AtmosConfigAbsolutePaths first (e.g. tests).
func getWorkflowsDirToUse(atmosConfig *schema.AtmosConfiguration) string {
if atmosConfig.WorkflowsDirAbsolutePath != "" {
return atmosConfig.WorkflowsDirAbsolutePath
}
// u.JoinPath (unlike filepath.Join) returns an already-absolute Workflows.BasePath as-is
// instead of nesting it under BasePath.
return u.JoinPath(atmosConfig.BasePath, atmosConfig.Workflows.BasePath)
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.

// enableProvenanceForRichOutput sets atmosConfig.TrackProvenance when the
// resolved output format is "rich". Provenance is enabled only for the rich
// invocation: it lets the command map JSON Schema fields back to the
Expand Down
Loading
Loading