Describe the Bug
metadata.labels/metadata.tags combined with the !labels/!tags YAML functions
(https://atmos.tools/changelog/tags-and-labels) are documented for exactly this use case:
The most common use is feeding metadata.labels into the var.tags a terraform-null-label-style module expects
This works correctly in isolation. It breaks the moment any other config layer also
contributes a value for the same vars.<key> — most notably terraform.overrides, commonly
used for a per-component/per-team single-key override on top of shared org-wide labels.
Instead of deep-merging, the other layer's value wholesale-replaces whatever
!labels/!tags would have produced, silently dropping every other key. No error, no
warning — atmos describe component ... --query .metadata.labels shows the full, correctly
merged label set the whole time, while .vars.tags (fed by !labels) shows only the
override's own key.
Isolated the exact trigger with two otherwise-identical scenarios (same team-defaults.yaml
override, same component, only the type of the org-level vars.tags default differs):
Org-level terraform.vars.tags default |
Result of atmos describe component comp -s prod --query .vars.tags |
Literal map: tags: {X: "1", Y: "2"} |
X: "1", Y: "2", Z: override-value — merged correctly |
YAML function: tags: !labels (backed by metadata.labels: {X: "1", Y: "2"}) — same effective values |
Z: override-value — X/Y silently gone |
Root cause, traced to source, not just symptoms: Atmos has a purpose-built mechanism to
solve exactly this class of problem — pkg/merge/merge_yaml_functions.go's
MergeWithDeferred/WalkAndDeferYAMLFunctions/ApplyDeferredMerges. When a YAML-function
string (e.g. "!template ...") needs to merge against a literal map from another layer,
naively merging would hit a type conflict (string vs. map) and whichever layer merges later
would just replace the string outright. So Atmos "defers" recognized YAML-function
strings — swapping them for a nil placeholder before the type-sensitive merge runs, then
resolving and merging them back in afterward.
Which functions get this treatment is a hardcoded list in isAtmosYAMLFunction:
// pkg/merge/merge_yaml_functions.go, lines 16-39
func isAtmosYAMLFunction(s string) bool {
if s == "" {
return false
}
// YAML functions processed after merging.
postMergeFunctions := []string{
"!template",
"!terraform.output",
"!terraform.state",
"!store.get",
"!store",
"!exec",
"!env",
}
for _, fn := range postMergeFunctions {
if strings.HasPrefix(s, fn) {
return true
}
}
return false
}
!labels, !tags, !labels.keys, and !labels.values are not in this list. So when
WalkAndDeferYAMLFunctions walks a vars map and finds tags: "!labels", it isn't
recognized as deferrable and passes through unchanged as a plain string. The merge that
follows — internal/exec/stack_processor_merge.go, mergeComponentConfigurations:
finalComponentVars, varsCtx, err := m.MergeWithDeferred(
mergeConfig,
[]map[string]any{
opts.GlobalVars,
result.BaseComponentVars,
result.ComponentVars,
result.ComponentOverridesVars,
},
)
then hits a genuine type conflict at that key: "!labels" (string) vs. whatever
ComponentOverridesVars.tags provides (a map). The only way to resolve a string-vs-map
conflict is for the higher-precedence layer to replace the lower one outright — so the
"!labels" marker is gone by the time the actual !labels function would otherwise run in
its own, separate, later pass. There's nothing left at that path to defer or evaluate against.
This also explains why it works with no other conflicting layer present at all: with nothing
else setting vars.tags, there's no type conflict to lose.
Expected Behavior
!labels/!tags/!labels.keys/!labels.values should merge with other layers the same way
!template, !env, !store, etc. already do — deep-merging into whatever any other layer
(especially overrides) contributes at that same path, not silently losing a type-conflict
race.
Steps to Reproduce
Minimal, self-contained repro. Creates a throwaway project in a temp dir, then compares the
same override against a literal-map default (works) and an equivalent !labels-produced
default (broken). Copy-paste the whole block into a terminal:
REPRO_DIR="$(mktemp -d)"
echo "Reproducing in: $REPRO_DIR"
# Install atmos if not already on PATH
if ! command -v atmos >/dev/null 2>&1; then
ATMOS_VERSION=1.225.0
ATMOS_BIN_DIR="$(mktemp -d)"
curl -fsSL "https://github.com/cloudposse/atmos/releases/download/v${ATMOS_VERSION}/atmos_${ATMOS_VERSION}_linux_amd64" -o "$ATMOS_BIN_DIR/atmos"
chmod +x "$ATMOS_BIN_DIR/atmos"
ATMOS="$ATMOS_BIN_DIR/atmos"
else
ATMOS=atmos
fi
$ATMOS version
cd "$REPRO_DIR"
git init -q
mkdir -p stacks components/terraform/comp
cat > atmos.yaml <<'EOF'
base_path: "./"
components:
terraform:
base_path: "components/terraform"
stacks:
base_path: "stacks"
included_paths: ["**/*"]
excluded_paths: ["**/_defaults.yaml", "**/team-defaults.yaml"]
name_pattern: "{stage}"
EOF
cat > components/terraform/comp/main.tf <<'EOF'
variable "tags" {
type = map(string)
default = {}
}
EOF
cat > stacks/team-defaults.yaml <<'EOF'
terraform:
overrides:
vars:
tags:
Z: override-value
EOF
echo "CASE A: org-level default is a literal map"
cat > stacks/_defaults.yaml <<'EOF'
terraform:
vars:
tags:
X: "1"
Y: "2"
EOF
cat > stacks/prod.yaml <<'EOF'
vars:
stage: prod
import: [_defaults, team-defaults]
components:
terraform:
comp: {}
EOF
$ATMOS describe component comp -s prod --query .vars.tags
echo ""
echo "CASE B: org-level default is !labels, backed by metadata.labels with the SAME effective values"
cat > stacks/_defaults.yaml <<'EOF'
metadata:
labels:
X: "1"
Y: "2"
terraform:
vars:
tags: !labels
EOF
$ATMOS describe component comp -s prod --query .vars.tags
Case A prints:
X: "1"
"Y": "2"
Z: override-value
Case B, identical override, only the default's mechanism changed, prints:
Screenshots
No screenshots — CLI output only, included in Steps to Reproduce above. The exact failure:
Case A correctly merges all three keys; Case B, with the same override, silently drops X
and Y and keeps only Z.
Environment
- OS: Linux
- Atmos version: 1.225.0 (also reproduces on
v1.226.0-rc.0, which is the same commit —
unfixed on main as of this writing)
- Standalone binary (linux/amd64), not run via Docker
- Config: minimal, single-file
atmos.yaml, no --config splitting, no unusual settings —
the conflict is purely between a terraform.vars type-level default and a
terraform.overrides.vars entry for the same key
Additional Context
Suggested fix direction: add the tags/labels function family to postMergeFunctions in
pkg/merge/merge_yaml_functions.go:
postMergeFunctions := []string{
"!template",
"!terraform.output",
"!terraform.state",
"!store.get",
"!store",
"!exec",
"!env",
"!labels",
"!labels.keys",
"!labels.values",
"!tags",
}
Worth double-checking the strings.HasPrefix ordering doesn't misclassify !labels.keys/
!labels.values as the bare !labels prefix (both start with !labels), but the fix is the
same shape as the six existing entries either way.
Impact: any stack using metadata.labels/metadata.tags + !labels/!tags — the
feature's own advertised use case — silently loses that data the moment any other config
layer also touches the same var, which for overrides (commonly paired with org-wide labels
for exactly this kind of per-component/per-team single-key override) is a very ordinary
configuration, not an edge case.
Relationship to existing issues: unrelated to #2867/#2868 (both about --config
file-splitting and CLI config re-initialization) — this reproduces with a single atmos.yaml
and no --config flag at all; the conflict is entirely within stack-manifest merging.
Describe the Bug
metadata.labels/metadata.tagscombined with the!labels/!tagsYAML functions(https://atmos.tools/changelog/tags-and-labels) are documented for exactly this use case:
This works correctly in isolation. It breaks the moment any other config layer also
contributes a value for the same
vars.<key>— most notablyterraform.overrides, commonlyused for a per-component/per-team single-key override on top of shared org-wide labels.
Instead of deep-merging, the other layer's value wholesale-replaces whatever
!labels/!tagswould have produced, silently dropping every other key. No error, nowarning —
atmos describe component ... --query .metadata.labelsshows the full, correctlymerged label set the whole time, while
.vars.tags(fed by!labels) shows only theoverride's own key.
Isolated the exact trigger with two otherwise-identical scenarios (same
team-defaults.yamloverride, same component, only the type of the org-level
vars.tagsdefault differs):terraform.vars.tagsdefaultatmos describe component comp -s prod --query .vars.tagstags: {X: "1", Y: "2"}X: "1", Y: "2", Z: override-value— merged correctlytags: !labels(backed bymetadata.labels: {X: "1", Y: "2"}) — same effective valuesZ: override-value—X/Ysilently goneRoot cause, traced to source, not just symptoms: Atmos has a purpose-built mechanism to
solve exactly this class of problem —
pkg/merge/merge_yaml_functions.go'sMergeWithDeferred/WalkAndDeferYAMLFunctions/ApplyDeferredMerges. When a YAML-functionstring (e.g.
"!template ...") needs to merge against a literal map from another layer,naively merging would hit a type conflict (string vs. map) and whichever layer merges later
would just replace the string outright. So Atmos "defers" recognized YAML-function
strings — swapping them for a
nilplaceholder before the type-sensitive merge runs, thenresolving and merging them back in afterward.
Which functions get this treatment is a hardcoded list in
isAtmosYAMLFunction:!labels,!tags,!labels.keys, and!labels.valuesare not in this list. So whenWalkAndDeferYAMLFunctionswalks avarsmap and findstags: "!labels", it isn'trecognized as deferrable and passes through unchanged as a plain string. The merge that
follows —
internal/exec/stack_processor_merge.go,mergeComponentConfigurations:then hits a genuine type conflict at that key:
"!labels"(string) vs. whateverComponentOverridesVars.tagsprovides (a map). The only way to resolve a string-vs-mapconflict is for the higher-precedence layer to replace the lower one outright — so the
"!labels"marker is gone by the time the actual!labelsfunction would otherwise run inits own, separate, later pass. There's nothing left at that path to defer or evaluate against.
This also explains why it works with no other conflicting layer present at all: with nothing
else setting
vars.tags, there's no type conflict to lose.Expected Behavior
!labels/!tags/!labels.keys/!labels.valuesshould merge with other layers the same way!template,!env,!store, etc. already do — deep-merging into whatever any other layer(especially
overrides) contributes at that same path, not silently losing a type-conflictrace.
Steps to Reproduce
Minimal, self-contained repro. Creates a throwaway project in a temp dir, then compares the
same override against a literal-map default (works) and an equivalent
!labels-produceddefault (broken). Copy-paste the whole block into a terminal:
Case A prints:
Case B, identical override, only the default's mechanism changed, prints:
Screenshots
No screenshots — CLI output only, included in Steps to Reproduce above. The exact failure:
Case A correctly merges all three keys; Case B, with the same override, silently drops
Xand
Yand keeps onlyZ.Environment
v1.226.0-rc.0, which is the same commit —unfixed on
mainas of this writing)atmos.yaml, no--configsplitting, no unusual settings —the conflict is purely between a
terraform.varstype-level default and aterraform.overrides.varsentry for the same keyAdditional Context
Suggested fix direction: add the tags/labels function family to
postMergeFunctionsinpkg/merge/merge_yaml_functions.go:Worth double-checking the
strings.HasPrefixordering doesn't misclassify!labels.keys/!labels.valuesas the bare!labelsprefix (both start with!labels), but the fix is thesame shape as the six existing entries either way.
Impact: any stack using
metadata.labels/metadata.tags+!labels/!tags— thefeature's own advertised use case — silently loses that data the moment any other config
layer also touches the same var, which for
overrides(commonly paired with org-wide labelsfor exactly this kind of per-component/per-team single-key override) is a very ordinary
configuration, not an edge case.
Relationship to existing issues: unrelated to #2867/#2868 (both about
--configfile-splitting and CLI config re-initialization) — this reproduces with a single
atmos.yamland no
--configflag at all; the conflict is entirely within stack-manifest merging.