scaffold generate --update collapses blank lines in YAML files that didn't actually change

[jorrite]

Atmos version

1.225.0

Reproduction

# 1. Generate a scaffold template with --git so it has a merge base.
atmos scaffold generate <template-with-a-yaml-file> ./out --defaults --git

# 2. Confirm the YAML file's blank lines are intact.
cat ./out/some-file.yaml

# 3. Re-run --update with nothing changed on the template side (same source, same ref).
atmos scaffold generate <same-template> ./out --defaults --update

# 4. Compare.
git -C ./out diff --stat   # the file shows a diff, despite no real change
cat ./out/some-file.yaml   # blank lines separating sections are gone

Expected

A YAML file with blank lines separating sections, run through --update with no meaningful change between the template's old and new versions, should come back byte-for-byte identical — blank lines included. --update should not rewrite a file it didn't actually change.

Impact

Every atmos scaffold generate --update run reformats YAML files that have no real changes, producing spurious diffs in the user's git history and destroying intentional visual structure (blank lines separating logical sections) in every managed YAML file, project-wide, on every update.

[jorrite]

Hit this in a scaffold template we are developing, destined for Azure DevOps. We bundle Azure Pipelines YAML (azure-pipelines.yml) as part of the generated project, and it's a very common convention in that ecosystem to use blank lines to visually separate jobs:/stages: entries and other top-level blocks, e.g. (literal copy-paste from MS doc examples):

  trigger:
  - main

  pool:
    vmImage: ubuntu-latest

  jobs:
  - job: PreWork
    steps:
    - script: "Do pre-work"

  - job: PostWork
    pool: windows-latest
    steps:
    - script: "Do post-work using a different hosted image"

Running the update procedure for already scaffolded projects removes all blank lines, even when nothing had really changed in either theirs/ours.

Spent a little time looking at possible directions here, ranging from a cheap short-circuit (skip re-encoding entirely when the merge produces no structural change, which covers the first case above) up to considering swapping the YAML library backing the 3-way merge for one with better round-trip/format-preserving behavior. The short-circuit approach only helps when nothing really changes, the moment something does, you just hit the issue again (obviously), so not really a great option. Swapping out the YAML library for something like https://github.com/goccy/go-yaml, but it would mean reimplementing the 3-way merge logic entirely.

[osterman]

I recall evaluating the goccy module, but can't entirely recall why we went with goyaml v4. I think it might be documented in the PRD, but I'm on my phone at the time and can't look it up. Bear in mind there are a lot of other considerations for deep merging YAML: comments, anchors, string encoding/representations, explicit tags, are a few.

An alternative might be to allow the user to configure that a scaffold force a text-based merge strategy, where whitespace is required for YAML.

[jorrite]

It's actually using the archived https://github.com/go-yaml/yaml/tree/v3.0.1, see:

atmos/pkg/generator/merge/yaml_merger.go

Line 11 in 3ce4349

"gopkg.in/yaml.v3"

Not sure I could find reasoning as to why it uses that over the YAML lib now maintained by https://github.com/yaml/go-yaml, but I explored v4 too and it doesn't solve the issue, same applies.

Also thought about first doing a pass with the text merger and if conflicts happen pass it through YAML instead...

Let me take up on the idea for a configurable driver mechanism for the YAML merge.

[osterman]

This feature was a beast. Actually started implementing it more than a year ago. Went through multiple reimplementations, rewrites, etc. I think it just got missed in the end. Ah wow, looks like v4 is still in RC stage, and to your point, doesn't solve the whitespace issue.

[osterman]

One other consideration, I've stood where you stand, having to deal with formatting and things getting reformatted. I prefer the formatting that you are adopting with your ADO pipelines.

That said, I've also had to deal with preserving/enforcing formatting at a team level. It gets very hard. And the only solution really comes down to having something that's programmatically enforceable and lintable. (Or just accept slightly less pleasing YAML formatting)

If we instead looked at this as a formatting issue (E.g. see atmos format) and solved it at a formatting layer, it might be another solution and kill 2 birds with one stone. We would still need to resolve how to achieve that.