feat: contrib Delta JVM claim/decline + scan marker [Delta contrib split, part 4a]#7
Draft
schenksj wants to merge 2 commits into
Draft
feat: contrib Delta JVM claim/decline + scan marker [Delta contrib split, part 4a]#7schenksj wants to merge 2 commits into
schenksj wants to merge 2 commits into
Conversation
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
38e2312 to
7c1773f
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
ae3ec8d to
2dc47b1
Compare
…lit, part 4a] Part 4a of the Delta Lake contrib PR breakup (tracking: apache#4366). The JVM claim/decline layer: `DeltaScanRule` recognises a V1 Delta scan and plants a `CometDeltaScanMarker` wrapping the original scan. This "activates" the reflective path A.2 already wired (`DeltaIntegration` looks up `DeltaScanRule$`), but stays INERT end to end: with no serde yet (`CometDeltaNativeScan`, part 4b), `CometExecRule`'s `scanHandler` lookup returns None, so the marker is left in the plan and executes as a vanilla Delta fallback. Net behavior on this build: Delta reads still run on vanilla Spark. - `DeltaScanRule.scala` - claim/decline rule (declines input_file_name, encryption, etc.; plants the marker otherwise). - `CometDeltaScanMarker.scala` - leaf exec node wrapping the original `FileSourceScanExec`; `doExecute` delegates to it (the vanilla fallback). FQN matches A.2's `DeltaIntegration. MarkerClass` string. - `DeltaScanMetadata.scala` - planning info carried on the marker; now also home to the `ScanImpl` constant (moved off the not-yet-present serde so the rule can name it). - `DeltaReflection.scala` - reflective Delta accessors (CDF members inert until part 5). - `RowTrackingAugmentedFileIndex.scala` - row-tracking file index used by the rule. - Tests: `CometDeltaTestBase` (trimmed of the serde/exec-dependent native-read helpers, which move to 4b; added marker-claim helpers) + a new `CometDeltaMarkerSuite` asserting the marker is planted on a plain read, the fallback is result-correct, and a declined projection plants no marker. - `dev/ci/check-suites.py` - exempt contrib test suites (they run under the dedicated Delta workflow, not the default pr_build matrix). No core / A.2 edits needed -- the reflective wiring already reaches this rule + marker. Required edit per the split plan: `CometDeltaNativeScan.ScanImpl` -> `DeltaScanMetadata.ScanImpl` (4b re-points the serde at it). Verification: gated JVM test-compile, `CometDeltaMarkerSuite` 3/3 (marker planted is red on the A.2 build / green here), check-suites, spotless + scalastyle, gate-verify (default build still 0 Delta symbols, only the DeltaIntegration bridge class) -- all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…P0+themeB DeltaReflection, folded into A.4a]
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
7c1773f to
0fab506
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
2dc47b1 to
dd9a0d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this part is
The JVM claim/decline layer.
DeltaScanRulerecognises a V1 Delta scan and plants aCometDeltaScanMarkerwrapping the original scan. This "activates" the reflective path part 2 already wired (DeltaIntegrationlooks upDeltaScanRule$), but stays inert end to end: with no serde yet (part 4b),CometExecRule'sscanHandlerlookup returnsNone, so the marker is left in the plan and executes as a vanilla Delta fallback. Net behavior on this build: Delta reads still run on vanilla Spark — so this is safe to land before the read path exists.DeltaScanRule— claim/decline (declinesinput_file_name, encryption, …; plants the marker otherwise).CometDeltaScanMarker— leaf exec wrapping the originalFileSourceScanExec;doExecutedelegates to it (the fallback). FQN matches part 2'sDeltaIntegration.MarkerClass.DeltaScanMetadata— planning info on the marker; now also home to theScanImplconstant (moved off the not-yet-present serde so the rule can name it; 4b re-points its serde here).DeltaReflection(CDF members inert until part 5),RowTrackingAugmentedFileIndex.CometDeltaTestBase(claim/marker helpers; the native-read helpers move to 4b) + a newCometDeltaMarkerSuite.dev/ci/check-suites.py— exempt contrib suites (they run under the dedicated Delta workflow).Why it's contained
No core / part-2 edits — the reflective wiring already reaches this rule + marker. A.4a touches only
contrib/delta/src+check-suites.py. The contrib Scala compiles only under-Pcontrib-delta; default builds carry zero Delta surface (gate-verify confirms 0 Delta symbols, only theDeltaIntegrationbridge class).Verification
gated JVM test-compile,
CometDeltaMarkerSuite3/3 (marker planted on a plain read — red on a part-2-only build, green here; fallback result-correct;input_file_name()declines with no marker), check-suites, spotless + scalastyle, and the gate-verify script — all green.🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.