Skip to content

!labels/!tags YAML functions are missing from the deferred-merge function list, so any other layer contributing the same map key silently replaces instead of merging #2888

Description

Describe the Bug

metadata.labels/metadata.tags combined with the !labels/!tags YAML functions
(https://atmos.tools/changelog/tags-and-labels) are documented for exactly this use case:

The most common use is feeding metadata.labels into the var.tags a terraform-null-label-style module expects

This works correctly in isolation. It breaks the moment any other config layer also
contributes a value for the same vars.<key> — most notably terraform.overrides, commonly
used for a per-component/per-team single-key override on top of shared org-wide labels.
Instead of deep-merging, the other layer's value wholesale-replaces whatever
!labels/!tags would have produced, silently dropping every other key. No error, no
warning — atmos describe component ... --query .metadata.labels shows the full, correctly
merged label set the whole time, while .vars.tags (fed by !labels) shows only the
override's own key.

Isolated the exact trigger with two otherwise-identical scenarios (same team-defaults.yaml
override, same component, only the type of the org-level vars.tags default differs):

Org-level terraform.vars.tags default Result of atmos describe component comp -s prod --query .vars.tags
Literal map: tags: {X: "1", Y: "2"} X: "1", Y: "2", Z: override-value — merged correctly
YAML function: tags: !labels (backed by metadata.labels: {X: "1", Y: "2"}) — same effective values Z: override-valueX/Y silently gone

Root cause, traced to source, not just symptoms: Atmos has a purpose-built mechanism to
solve exactly this class of problem — pkg/merge/merge_yaml_functions.go's
MergeWithDeferred/WalkAndDeferYAMLFunctions/ApplyDeferredMerges. When a YAML-function
string (e.g. "!template ...") needs to merge against a literal map from another layer,
naively merging would hit a type conflict (string vs. map) and whichever layer merges later
would just replace the string outright. So Atmos "defers" recognized YAML-function
strings — swapping them for a nil placeholder before the type-sensitive merge runs, then
resolving and merging them back in afterward.

Which functions get this treatment is a hardcoded list in isAtmosYAMLFunction:

// pkg/merge/merge_yaml_functions.go, lines 16-39
func isAtmosYAMLFunction(s string) bool {
	if s == "" {
		return false
	}

	// YAML functions processed after merging.
	postMergeFunctions := []string{
		"!template",
		"!terraform.output",
		"!terraform.state",
		"!store.get",
		"!store",
		"!exec",
		"!env",
	}

	for _, fn := range postMergeFunctions {
		if strings.HasPrefix(s, fn) {
			return true
		}
	}

	return false
}

!labels, !tags, !labels.keys, and !labels.values are not in this list. So when
WalkAndDeferYAMLFunctions walks a vars map and finds tags: "!labels", it isn't
recognized as deferrable and passes through unchanged as a plain string. The merge that
follows — internal/exec/stack_processor_merge.go, mergeComponentConfigurations:

finalComponentVars, varsCtx, err := m.MergeWithDeferred(
	mergeConfig,
	[]map[string]any{
		opts.GlobalVars,
		result.BaseComponentVars,
		result.ComponentVars,
		result.ComponentOverridesVars,
	},
)

then hits a genuine type conflict at that key: "!labels" (string) vs. whatever
ComponentOverridesVars.tags provides (a map). The only way to resolve a string-vs-map
conflict is for the higher-precedence layer to replace the lower one outright — so the
"!labels" marker is gone by the time the actual !labels function would otherwise run in
its own, separate, later pass. There's nothing left at that path to defer or evaluate against.

This also explains why it works with no other conflicting layer present at all: with nothing
else setting vars.tags, there's no type conflict to lose.

Expected Behavior

!labels/!tags/!labels.keys/!labels.values should merge with other layers the same way
!template, !env, !store, etc. already do — deep-merging into whatever any other layer
(especially overrides) contributes at that same path, not silently losing a type-conflict
race.

Steps to Reproduce

Minimal, self-contained repro. Creates a throwaway project in a temp dir, then compares the
same override against a literal-map default (works) and an equivalent !labels-produced
default (broken). Copy-paste the whole block into a terminal:

REPRO_DIR="$(mktemp -d)"
echo "Reproducing in: $REPRO_DIR"

# Install atmos if not already on PATH
if ! command -v atmos >/dev/null 2>&1; then
  ATMOS_VERSION=1.225.0
  ATMOS_BIN_DIR="$(mktemp -d)"
  curl -fsSL "https://github.com/cloudposse/atmos/releases/download/v${ATMOS_VERSION}/atmos_${ATMOS_VERSION}_linux_amd64" -o "$ATMOS_BIN_DIR/atmos"
  chmod +x "$ATMOS_BIN_DIR/atmos"
  ATMOS="$ATMOS_BIN_DIR/atmos"
else
  ATMOS=atmos
fi
$ATMOS version

cd "$REPRO_DIR"
git init -q
mkdir -p stacks components/terraform/comp

cat > atmos.yaml <<'EOF'
base_path: "./"
components:
  terraform:
    base_path: "components/terraform"
stacks:
  base_path: "stacks"
  included_paths: ["**/*"]
  excluded_paths: ["**/_defaults.yaml", "**/team-defaults.yaml"]
  name_pattern: "{stage}"
EOF

cat > components/terraform/comp/main.tf <<'EOF'
variable "tags" {
  type    = map(string)
  default = {}
}
EOF

cat > stacks/team-defaults.yaml <<'EOF'
terraform:
  overrides:
    vars:
      tags:
        Z: override-value
EOF

echo "CASE A: org-level default is a literal map"
cat > stacks/_defaults.yaml <<'EOF'
terraform:
  vars:
    tags:
      X: "1"
      Y: "2"
EOF
cat > stacks/prod.yaml <<'EOF'
vars:
  stage: prod
import: [_defaults, team-defaults]
components:
  terraform:
    comp: {}
EOF
$ATMOS describe component comp -s prod --query .vars.tags

echo ""
echo "CASE B: org-level default is !labels, backed by metadata.labels with the SAME effective values"
cat > stacks/_defaults.yaml <<'EOF'
metadata:
  labels:
    X: "1"
    Y: "2"
terraform:
  vars:
    tags: !labels
EOF
$ATMOS describe component comp -s prod --query .vars.tags

Case A prints:

X: "1"
"Y": "2"
Z: override-value

Case B, identical override, only the default's mechanism changed, prints:

Z: override-value

Screenshots

No screenshots — CLI output only, included in Steps to Reproduce above. The exact failure:
Case A correctly merges all three keys; Case B, with the same override, silently drops X
and Y and keeps only Z.

Environment

  • OS: Linux
  • Atmos version: 1.225.0 (also reproduces on v1.226.0-rc.0, which is the same commit —
    unfixed on main as of this writing)
  • Standalone binary (linux/amd64), not run via Docker
  • Config: minimal, single-file atmos.yaml, no --config splitting, no unusual settings —
    the conflict is purely between a terraform.vars type-level default and a
    terraform.overrides.vars entry for the same key

Additional Context

Suggested fix direction: add the tags/labels function family to postMergeFunctions in
pkg/merge/merge_yaml_functions.go:

postMergeFunctions := []string{
	"!template",
	"!terraform.output",
	"!terraform.state",
	"!store.get",
	"!store",
	"!exec",
	"!env",
	"!labels",
	"!labels.keys",
	"!labels.values",
	"!tags",
}

Worth double-checking the strings.HasPrefix ordering doesn't misclassify !labels.keys/
!labels.values as the bare !labels prefix (both start with !labels), but the fix is the
same shape as the six existing entries either way.

Impact: any stack using metadata.labels/metadata.tags + !labels/!tags — the
feature's own advertised use case — silently loses that data the moment any other config
layer also touches the same var, which for overrides (commonly paired with org-wide labels
for exactly this kind of per-component/per-team single-key override) is a very ordinary
configuration, not an edge case.

Relationship to existing issues: unrelated to #2867/#2868 (both about --config
file-splitting and CLI config re-initialization) — this reproduces with a single atmos.yaml
and no --config flag at all; the conflict is entirely within stack-manifest merging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions