ci: add Scalpel shadow comparison and upgrade to 0.3.7

.github/CI-ARCHITECTURE.md

@@ -154,12 +154,30 @@ Both methods run in parallel. Results are merged (union) before testing. This le
 2. **No regression** — If Scalpel fails, grep results are still used
 3. **Gradual migration** — Once Scalpel is validated, grep can be removed
 
-Scalpel is configured permanently in `.mvn/extensions.xml` (version `0.1.0`). On developer machines it is a no-op — without CI environment variables (`GITHUB_BASE_REF`), no base branch is detected and Scalpel returns immediately. The `mvn validate` with report mode adds ~60-90 seconds in CI.
-
-Note: the script overrides `fullBuildTriggers` to empty (`-Dscalpel.fullBuildTriggers=`) because Scalpel's default (`.mvn/**`) would trigger a full build whenever `.mvn/extensions.xml` itself changes (e.g., Dependabot bumping Scalpel).
+Scalpel is configured permanently in `.mvn/extensions.xml`. On developer machines it is a no-op (disabled via `-Dscalpel.enabled=false` in `.mvn/maven.config`). The CI script overrides this with `-Dscalpel.enabled=true`. The `mvn validate` with report mode adds ~60-90 seconds in CI.
 
 Scalpel is only invoked when a **subdirectory** `pom.xml` is changed (e.g. `parent/pom.xml`, `components/camel-kafka/pom.xml`). Changes to the **root** `pom.xml` are excluded because it contains build-infrastructure config (license plugin, checkstyle, etc.) that does not affect module compilation or test behavior. Without this filter, Scalpel would report every module as affected since they all inherit from the root POM.
 
+#### Scalpel features used for shadow comparison
+
+- **Source-set-aware propagation**: Distinguishes test-jar dependencies from regular dependencies. A module that depends only on another module's test-jar (e.g., `camel-core`'s test-jar with test utilities) is propagated through the `TEST` source set, not the `MAIN` source set. This prevents a change to test utilities from triggering tests in all ~500 modules that depend on `camel-core`.
+- **`skipTestsForDownstreamModules`**: Allows specifying modules whose tests should be skipped when they appear as downstream dependents (mirrors the `EXCLUSION_LIST` in `incremental-build.sh`). This gives Scalpel an accurate picture of what skip-tests mode would actually test.
+
+#### Shadow comparison
+
+Scalpel runs in **shadow mode**: it observes what skip-tests mode *would* have done and reports it in a collapsible section of the PR comment, without affecting actual test execution. This allows the team to validate Scalpel's decisions across many PRs before switching to Scalpel-driven test execution.
+
+The shadow comparison section shows:
+- How many modules Scalpel would test (direct + downstream)
+- How many downstream modules would have tests skipped (generated code, meta-modules)
+- The full list of modules in each category
+
+#### Configuration notes
+
+The script overrides `fullBuildTriggers` to empty (`-Dscalpel.fullBuildTriggers=`) because Scalpel's default (`.mvn/**`) would trigger a full build whenever `.mvn/extensions.xml` itself changes (e.g., Dependabot bumping Scalpel).
+
+The grep-based script fetches the PR diff via the GitHub REST API (unchanged). Scalpel uses local git history to compare effective POM models — the CI workflow pre-fetches the base branch (`git fetch --deepen=200` + fetch of `origin/main`) so Scalpel's JGit can find the merge-base. Scalpel disables its built-in JGit fetch (`-Dscalpel.fetchBaseBranch=false`) to avoid JGit issues in shallow CI clones. The `--deepen=200` fetches only commit metadata (not file blobs), adding ~2-3 seconds to the job.
+
 ## Manual Integration Test Advisories
 
 Some modules are excluded from CI's `-amd` expansion (the `EXCLUSION_LIST`) because they are generated code, meta-modules, or expensive integration test suites. When a contributor changes one of these modules, CI cannot automatically test all downstream effects.

.github/actions/incremental-build/incremental-build.sh

@@ -243,36 +243,49 @@ analyzePomDependencies() {
 runScalpelDetection() {
   echo "  Running Scalpel change detection..."
 
-  # Ensure sufficient git history for JGit merge-base detection
-  # (CI uses shallow clones; Scalpel needs to find the merge base)
-  git fetch origin main:refs/remotes/origin/main --depth=200 2>/dev/null || true
-  git fetch --deepen=200 2>/dev/null || true
-
   # Scalpel is permanently configured in .mvn/extensions.xml.
-  # On developer machines it's a no-op (no GITHUB_BASE_REF → no base branch detected).
+  # On developer machines it's a no-op (disabled via -Dscalpel.enabled=false in .mvn/maven.config).
+  # The CI script overrides this with -Dscalpel.enabled=true.
+  # Base branch is pre-fetched by the CI workflow (fetchBaseBranch=false).
   # Run Maven validate with Scalpel in report mode:
   # - mode=report: write JSON report without trimming the reactor
   # - fullBuildTriggers="": override .mvn/** default (Scalpel lives in .mvn/extensions.xml)
-  # - alsoMake/alsoMakeDependents=false: we only want directly affected modules
-  #   (our script handles -amd expansion separately)
-  local scalpel_args="-Dscalpel.enabled=true -Dscalpel.mode=report -Dscalpel.fullBuildTriggers= -Dscalpel.alsoMake=false -Dscalpel.alsoMakeDependents=false"
-  # For workflow_dispatch, GITHUB_BASE_REF may not be set
-  if [ -z "${GITHUB_BASE_REF:-}" ]; then
-    scalpel_args="$scalpel_args -Dscalpel.baseBranch=origin/main"
+  # - fetchBaseBranch=false: base branch is pre-fetched by the CI workflow
+  # - skipTestsForDownstreamModules: derived from EXCLUSION_LIST — tells Scalpel which
+  #   downstream modules should not run tests in skip-tests mode (for shadow comparison)
+  # Strip the Maven "!:" prefix from each entry to get bare artifact IDs for Scalpel.
+  local skip_downstream
+  skip_downstream=$(echo "$EXCLUSION_LIST" | sed 's/!://g')
+  # Always pass baseBranch explicitly — relying on Scalpel's env.GITHUB_BASE_REF
+  # auto-detection is fragile across Maven wrappers and CI rerun contexts.
+  local base_branch="origin/${GITHUB_BASE_REF:-main}"
+  local scalpel_args="-Dscalpel.enabled=true -Dscalpel.mode=report -Dscalpel.fullBuildTriggers= -Dscalpel.fetchBaseBranch=false -Dscalpel.baseBranch=${base_branch} -Dscalpel.excludePaths=.github/** -Dscalpel.skipTestsForDownstreamModules=${skip_downstream}"
+
+  # Verify merge base is reachable before running Scalpel
+  if ! git merge-base HEAD "${base_branch}" >/dev/null 2>&1; then
+    echo "  WARNING: merge base between HEAD and ${base_branch} is not reachable"
+    echo "  HEAD=$(git rev-parse HEAD 2>/dev/null), ${base_branch}=$(git rev-parse ${base_branch} 2>/dev/null || echo 'NOT FOUND')"
+    scalpel_failure_reason="Merge base not reachable between HEAD and ${base_branch} (shallow clone too shallow?)"
+    return
   fi
 
-  echo "  Scalpel: running mvn validate (report mode)..."
+  echo "  Scalpel: running mvn validate (report mode, base=${base_branch})..."
   ./mvnw -B -q validate $scalpel_args ${MAVEN_EXTRA_ARGS:-} -l /tmp/scalpel-validate.log 2>/dev/null || {
     echo "  WARNING: Scalpel detection failed (exit $?), skipping"
     grep -i "scalpel" /tmp/scalpel-validate.log 2>/dev/null | head -5 || true
+    scalpel_failure_reason="Scalpel detection failed (mvn validate exited with error)"
     return
   }
 
   # Parse the Scalpel report
   local report="target/scalpel-report.json"
   if [ ! -f "$report" ]; then
     echo "  WARNING: Scalpel report not found at $report"
-    grep -i "scalpel" /tmp/scalpel-validate.log 2>/dev/null | head -5 || true
+    echo "  Scalpel log (last 10 lines):"
+    tail -10 /tmp/scalpel-validate.log 2>/dev/null || true
+    echo "  Scalpel-specific messages:"
+    grep -i "scalpel\|merge.base\|JGit\|no changes" /tmp/scalpel-validate.log 2>/dev/null | head -10 || true
+    scalpel_failure_reason="Scalpel report not found (merge-base may be unreachable in shallow clone)"
     return
   fi
 
@@ -283,6 +296,7 @@ runScalpelDetection() {
     local trigger_file
     trigger_file=$(jq -r '.triggerFile // "unknown"' "$report")
     echo "  Scalpel: Full build triggered by change to $trigger_file"
+    scalpel_failure_reason="Scalpel triggered a full build (changed file: $trigger_file)"
     return
   fi
 
@@ -293,9 +307,24 @@ runScalpelDetection() {
   scalpel_managed_deps=$(jq -r '(.changedManagedDependencies // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true)
   scalpel_managed_plugins=$(jq -r '(.changedManagedPlugins // []) | if length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true)
 
+  # Scalpel shadow comparison data:
+  # - Modules Scalpel skip-tests mode would test (testsSkipped != true)
+  # - Modules Scalpel would skip (testsSkipped == true, from skipTestsForDownstreamModules)
+  # - Breakdown by category (DIRECT, DOWNSTREAM)
+  scalpel_would_test=$(jq -r '[.affectedModules[] | select(.testsSkipped != true)] | map(.artifactId) | sort | join(",")' "$report" 2>/dev/null || true)
+  scalpel_would_skip=$(jq -r '[.affectedModules[] | select(.testsSkipped == true)] | map(.artifactId) | sort | join(",")' "$report" 2>/dev/null || true)
+  scalpel_direct_count=$(jq '[.affectedModules[] | select(.category == "DIRECT")] | length' "$report" 2>/dev/null || echo "0")
+  scalpel_downstream_tested=$(jq '[.affectedModules[] | select(.category == "DOWNSTREAM" and .testsSkipped != true)] | length' "$report" 2>/dev/null || echo "0")
+  scalpel_downstream_skipped=$(jq '[.affectedModules[] | select(.category == "DOWNSTREAM" and .testsSkipped == true)] | length' "$report" 2>/dev/null || echo "0")
+
   local mod_count
   mod_count=$(jq '.affectedModules | length' "$report" 2>/dev/null || echo "0")
-  echo "  Scalpel detected $mod_count affected modules"
+  local test_count=0
+  if [ -n "$scalpel_would_test" ]; then
+    test_count=$(echo "$scalpel_would_test" | tr ',' '\n' | grep -c . || true)
+  fi
+  echo "  Scalpel detected $mod_count affected modules ($test_count would be tested)"
+  echo "    Direct: $scalpel_direct_count, Downstream tested: $scalpel_downstream_tested, Downstream skipped: $scalpel_downstream_skipped"
   if [ -n "$scalpel_props" ]; then
     echo "    Changed properties: $scalpel_props"
   fi
@@ -382,6 +411,81 @@ checkManualItTests() {
   fi
 }
 
+# ── Scalpel shadow comparison ──────────────────────────────────────────
+
+# Write Scalpel shadow comparison section to the PR comment.
+# Shows what Scalpel skip-tests mode would have tested vs what the current
+# approach actually tested — observation only, does not affect test execution.
+writeScalpelComparison() {
+  local comment_file="$1"
+
+  # If Scalpel failed, show why in the PR comment
+  if [ -n "$scalpel_failure_reason" ]; then
+    echo "" >> "$comment_file"
+    echo "<details><summary>:microscope: Scalpel shadow comparison (skip-tests mode)</summary>" >> "$comment_file"
+    echo "" >> "$comment_file"
+    echo ":warning: $scalpel_failure_reason" >> "$comment_file"
+    echo "" >> "$comment_file"
+    echo "> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)" >> "$comment_file"
+    echo "" >> "$comment_file"
+    echo "</details>" >> "$comment_file"
+    return
+  fi
+
+  # Skip if no Scalpel data (Scalpel was not invoked for this PR)
+  if [ -z "$scalpel_would_test" ] && [ -z "$scalpel_would_skip" ]; then
+    return
+  fi
+
+  local scalpel_test_count=0
+  local scalpel_skip_count=0
+  if [ -n "$scalpel_would_test" ]; then
+    scalpel_test_count=$(echo "$scalpel_would_test" | tr ',' '\n' | grep -c . || true)
+  fi
+  if [ -n "$scalpel_would_skip" ]; then
+    scalpel_skip_count=$(echo "$scalpel_would_skip" | tr ',' '\n' | grep -c . || true)
+  fi
+
+  echo "" >> "$comment_file"
+  echo "<details><summary>:microscope: Scalpel shadow comparison (skip-tests mode)</summary>" >> "$comment_file"
+  echo "" >> "$comment_file"
+  echo "**Scalpel skip-tests mode would test ${scalpel_test_count} modules** (${scalpel_direct_count} direct + ${scalpel_downstream_tested} downstream)" >> "$comment_file"
+
+  if [ "$scalpel_downstream_skipped" -gt 0 ]; then
+    echo "" >> "$comment_file"
+    echo "${scalpel_downstream_skipped} downstream module(s) would have tests skipped (generated code, meta-modules)" >> "$comment_file"
+  fi
+
+  # Show which modules Scalpel would test
+  if [ -n "$scalpel_would_test" ]; then
+    echo "" >> "$comment_file"
+    echo "<details><summary>Modules Scalpel would test (${scalpel_test_count})</summary>" >> "$comment_file"
+    echo "" >> "$comment_file"
+    echo "$scalpel_would_test" | tr ',' '\n' | while read -r m; do
+      [ -n "$m" ] && echo "- \`$m\`" >> "$comment_file"
+    done
+    echo "" >> "$comment_file"
+    echo "</details>" >> "$comment_file"
+  fi
+
+  # Show which modules would have tests skipped
+  if [ -n "$scalpel_would_skip" ]; then
+    echo "" >> "$comment_file"
+    echo "<details><summary>Modules with tests skipped (${scalpel_skip_count})</summary>" >> "$comment_file"
+    echo "" >> "$comment_file"
+    echo "$scalpel_would_skip" | tr ',' '\n' | while read -r m; do
+      [ -n "$m" ] && echo "- \`$m\`" >> "$comment_file"
+    done
+    echo "" >> "$comment_file"
+    echo "</details>" >> "$comment_file"
+  fi
+
+  echo "" >> "$comment_file"
+  echo "> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)" >> "$comment_file"
+  echo "" >> "$comment_file"
+  echo "</details>" >> "$comment_file"
+}
+
 # ── Comment generation ─────────────────────────────────────────────────
 
 writeComment() {
@@ -539,6 +643,13 @@ main() {
   scalpel_props=""
   scalpel_managed_deps=""
   scalpel_managed_plugins=""
+  # Scalpel shadow comparison data
+  scalpel_would_test=""
+  scalpel_would_skip=""
+  scalpel_direct_count="0"
+  scalpel_downstream_tested="0"
+  scalpel_downstream_skipped="0"
+  scalpel_failure_reason=""
 
   # Step 2a: Grep-based detection (existing approach)
   if [ -n "$pom_files" ]; then
@@ -762,6 +873,9 @@ main() {
   local comment_file="incremental-test-comment.md"
   writeComment "$comment_file" "$pl" "$dep_module_ids" "$all_changed_props" "$testedDependents" "$extraModules" "$scalpel_managed_deps" "$scalpel_managed_plugins"
 
+  # Scalpel shadow comparison (observation only)
+  writeScalpelComparison "$comment_file"
+
   # Check for tests disabled in CI via @DisabledIfSystemProperty(named = "ci.env.name")
   local disabled_tests
   disabled_tests=$(detectDisabledTests "$final_pl")

.github/workflows/pr-build-main.yml

@@ -80,6 +80,29 @@ jobs:
         with:
           persist-credentials: false
           ref: ${{ inputs.pr_ref || '' }}
+      - name: Fetch base branch for Scalpel change detection
+        if: ${{ !inputs.skip_full_build }}
+        run: |
+          # Scalpel needs the merge base between HEAD and the base branch.
+          # The checkout is depth=1, so we progressively deepen until the
+          # merge base is reachable: 200 → 1000 → full history.
+          # Scalpel is observational — fetch failures must not break the build.
+          BASE_REF="${GITHUB_BASE_REF:-main}"
+          for depth in 200 1000; do
+            git fetch --deepen=$depth 2>/dev/null || true
+            git fetch --no-tags --depth=$depth origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true
+            if git merge-base HEAD "origin/${BASE_REF}" >/dev/null 2>&1; then
+              echo "Merge base reachable at depth $depth"
+              break
+            fi
+            echo "Merge base not reachable at depth $depth, deepening..."
+          done
+          # If still not reachable, fetch full history as last resort
+          if ! git merge-base HEAD "origin/${BASE_REF}" >/dev/null 2>&1; then
+            echo "Merge base still not reachable, fetching full history"
+            git fetch --unshallow 2>/dev/null || true
+            git fetch --no-tags origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true
+          fi
       - id: install-packages
         uses: ./.github/actions/install-packages
       - id: install-mvnd
@@ -148,8 +171,11 @@ jobs:
           name: incremental-test-java-${{ matrix.java }}.log
           path: incremental-test.log
       # All non-experimental JDK matrix entries upload with overwrite: true.
-      # This ensures a comment is posted even if one JDK build fails — the
-      # content is identical across JDKs (same modules tested), so last writer wins.
+      # The comment content is identical across JDKs (same modules tested),
+      # so last writer wins. However, we only upload (and overwrite) when the
+      # comment file actually exists — a cancelled build (e.g., JDK 25 killed
+      # while JDK 17 fails) won't have the file and must not overwrite an
+      # artifact from a matrix entry that did produce it.
       - name: Save PR number and test comment for commenter workflow
         if: always() && !matrix.experimental
         shell: bash
@@ -167,7 +193,9 @@ jobs:
             echo ":gear: [View full build and test results](${RUN_URL})" >> ci-comment-artifact/incremental-test-comment.md
           fi
       - name: Upload CI comment artifact
-        if: always() && !matrix.experimental
+        if: >
+          always() && !matrix.experimental &&
+          hashFiles('ci-comment-artifact/incremental-test-comment.md') != ''
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: ci-comment

.github/workflows/sonar-build.yml

@@ -45,7 +45,25 @@ jobs:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
-
+      - name: Fetch base branch for Scalpel change detection
+        run: |
+          # Scalpel is observational — fetch failures must not break the build.
+          BASE_REF="${GITHUB_BASE_REF:-main}"
+          for depth in 200 1000; do
+            git fetch --deepen=$depth 2>/dev/null || true
+            git fetch --no-tags --depth=$depth origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true
+            if git merge-base HEAD "origin/${BASE_REF}" >/dev/null 2>&1; then
+              echo "Merge base reachable at depth $depth"
+              break
+            fi
+            echo "Merge base not reachable at depth $depth, deepening..."
+          done
+          # If still not reachable, fetch full history as last resort
+          if ! git merge-base HEAD "origin/${BASE_REF}" >/dev/null 2>&1; then
+            echo "Merge base still not reachable, fetching full history"
+            git fetch --unshallow 2>/dev/null || true
+            git fetch --no-tags origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true
+          fi
       - id: install-packages
         uses: ./.github/actions/install-packages
 

.mvn/extensions.xml

@@ -23,6 +23,6 @@
     <extension>
         <groupId>eu.maveniverse.maven.scalpel</groupId>
         <artifactId>extension3</artifactId>
-        <version>0.3.5</version>
+        <version>0.3.7</version>
     </extension>
 </extensions>