feat(globals): add per-property merge strategy for list properties#3945
feat(globals): add per-property merge strategy for list properties#3945vicheey wants to merge 8 commits into
Conversation
Add a strategy pattern to the Globals merge engine that allows declaring per-property merge behavior via a dot-notation schema. Day-1 scope: Function.Architectures uses REPLACE (local list wins entirely). The engine remains fully backward-compatible: properties not listed in CUSTOM_STRATEGIES default to CONCATENATE (today's behavior). Path-aware recursion tracks the current property path and consults the schema only at LIST+LIST merge nodes. New module: samtranslator/plugins/globals/merge_strategy.py - MergeOp enum: CONCATENATE, REPLACE, MERGE_BY_KEY - MergeRule frozen dataclass with validation - merge_by_key() factory function Resolves: #3939
…cate keys Guard against duplicate key values in global_list (produces duplicate replacements) and local_list (produces duplicate appends) by checking seen_keys before appending in both passes. Adds regression tests for both cases.
|
This PR takes inspiration from #3940 (by @allenheltondev) and expands the approach into a general-purpose framework with a strategy pattern that supports:
The framework is extensible via cc @allenheltondev — thank you for the original fix and for raising #3939. |
…irements.Architectures Extends CUSTOM_STRATEGIES with a nested dot-path rule proving the framework works at 2+ levels of dict recursion. Adds test cases to both the dedicated merge strategy fixture and the existing capacity_provider_global_with_functions fixture.
| :param local_list: Local list of dicts | ||
| :param key: The dict key to match on | ||
| :return: Merged list | ||
| """ |
There was a problem hiding this comment.
[BUG] _merge_by_key produces inconsistent results when local_list contains duplicate key values, depending on whether global_list happens to contain the same key.
local_by_key = {item[key]: item for item in local_list if isinstance(item, dict) and key in item}The dict comprehension causes the last local entry to win for any given key. Then in Pass 2, the loop iterates local_list directly and appends the first non-seen entry. The two passes disagree:
- Global has no override for the duplicate key → Pass 2 wins → first local entry kept.
- Global has an override for the duplicate key → Pass 1 wins via local_by_key → last local entry kept.
Trace with key="Key":
# Case A: no global override
global_list = []
local_list = [{"Key": "env", "Value": "a"}, {"Key": "env", "Value": "b"}]
# result -> [{"Key": "env", "Value": "a"}] (first wins)
# Case B: global has the same key
global_list = [{"Key": "env", "Value": "g"}]
local_list = [{"Key": "env", "Value": "a"}, {"Key": "env", "Value": "b"}]
# result -> [{"Key": "env", "Value": "b"}] (last wins, via local_by_key)The existing test list_with_merge_by_key_deduplicates_local_duplicates covers Case A and asserts first-wins; Case B is untested and silently flips the precedence. Recommend picking one rule and using it in both passes — e.g. build local_by_key by skipping already-present keys (if item[key] not in local_by_key) so first-wins is consistent across both branches, or by iterating local_by_key.values() in Pass 2 so last-wins is consistent.
This is dormant for now (no CUSTOM_STRATEGIES entry uses MERGE_BY_KEY), but the test suite locks in the inconsistent behavior, which will be harder to change once Tags merge-by-key is enabled.
…sses local_by_key dict comprehension kept last entry per key, but Pass 2 kept first non-seen entry. When global had the same key as duplicate locals, Pass 1 used last-wins (via local_by_key) while Pass 2 used first-wins — inconsistent behavior depending on whether global contained the key. Fix: build local_by_key with first-entry-wins (skip already-present keys) so both passes agree. Adds regression test for Case B.
…ties
Adds a new merge strategy that replaces at the key level (only local's
keys survive) but deep-merges values when both global and local share
the same key. No parameters needed — existing recursion handles all
value types (dicts deep-merge, scalars local-win, lists concatenate).
Registered for CapacityProvider.ManagedResourceTags: when local sets
Tags, global Propagate is dropped; when both have Tags, values merge.
Empty local {} inherits global (falsy guard, matches SAM precedent).
…urceTags - New MergeOp: local key-set wins, shared keys deep-merge values - Intercept DICT+DICT nodes in _do_merge when strategy registered - Add CapacityProvider.ManagedResourceTags to CUSTOM_STRATEGIES - Fix ruff PLC0206 (.items()) and mypy no-untyped-call - Update translator fixtures: CpOverrideStrategyDropsGlobalKey added, CpExplicitTagsConflictWithGlobal removed (no longer an error)
…bals-merge-strategy
Summary
Fixes #3939
Adds a per-property merge strategy to the Globals merge engine. Today, all list-type properties concatenate when both Globals and resource-level values exist. This is incorrect for properties like
Architectures(a function runs on one architecture — local should replace, not append).Changes
New module:
samtranslator/plugins/globals/merge_strategy.pyMergeOpenum:CONCATENATE,REPLACE,MERGE_BY_KEYMergeRulefrozen dataclass with validationmerge_by_key()factory functionModified:
samtranslator/plugins/globals/globals.py_do_mergetracks the current property path through dict recursionCUSTOM_STRATEGIESdict declares per-property behavior using dot-notation paths_merge_by_key()method for future use (Tags merge-by-key)Day-1 scope:
Function.Architectures: REPLACEDesign
The schema uses dot-notation paths (e.g.,
VpcConfig.SecurityGroupIds) to support nested properties. Properties not listed inCUSTOM_STRATEGIESdefault toCONCATENATE— identical to today's behavior. This makes the change fully backward-compatible.How to extend
Add entries to
CUSTOM_STRATEGIESinglobals.py:Tests
MergeRuletypes and validation (12 cases)GlobalPropertiesTestCasescovering REPLACE, MERGE_BY_KEY, nested paths, and multi-strategyglobals_merge_strategy_architectures.yaml) proving REPLACE behavior across all 3 partitions