feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2]#4
Draft
schenksj wants to merge 1 commit into
Draft
feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2]#4schenksj wants to merge 1 commit into
schenksj wants to merge 1 commit into
Conversation
…b split, part 2] Part 2 of the Delta Lake contrib PR breakup (tracking: apache#4366). Establishes the `contrib-delta` build gate and the inert wiring that lets a gated build compile end to end, while the DEFAULT build stays byte-for-byte unchanged (zero Delta surface). No real Delta read logic yet -- that lands in later parts; here a Delta read that reaches native returns a clean "not implemented" error and falls back to vanilla Spark. Build gate: - Maven `contrib-delta` profile (spark/pom.xml) with per-Spark `delta.version` (3.5->3.3.2, 4.0->4.0.0, 4.1->4.1.0) and an add-source of contrib/delta/src. Default `delta.version` floor in pom.xml. The default spark.version stays 4.1.2 (the delta-spark 4.1.1 pin is a separate, deferred decision). - Cargo `contrib-delta` feature on core (optional path dep on comet-contrib-delta); `native/Cargo.toml` excludes ../contrib from the workspace. - `dev/verify-contrib-delta-gate.sh` proves default cargo/mvn/dylib carry zero Delta surface and the gated build pulls the right deps; wired into a minimal `delta_build_gate.yml` CI job (the full suite/regression workflows land later). Hardened the script against a `set -o pipefail` + `grep -q` SIGPIPE misfire (early grep exit -> echo SIGPIPE -> false guard failure) via here-strings. Inert wiring: - Proto: `Delta*` messages + `delta_scan = 118` (117 is BroadcastNestedLoopJoin). - Native dispatch: `OpStruct::DeltaScan` arm with a not-compiled-in error on default builds and a feature-gated `delta_scan` shim that calls the contrib; exhaustive-match arms in operator_registry/jni_api; `convert_spark_types_to_ arrow_schema` promoted to pub(crate). - Stub contrib crate (contrib/delta/native): `plan_delta_scan` returns `DataFusionError::NotImplemented` -- just enough to satisfy the core shim's contract so `--features contrib-delta` links. - JVM bridge `DeltaIntegration` (reflective, all lookups return None until the contrib classes exist), the CometExecRule Delta-marker hook (CDF hook deferred to a later part), the CometScanRule Delta delegation + metadata-col reorder, and the leaf `DeltaConf`. Verification: default + gated native build, clippy both feature states, gate script, gated + default JVM compile, spotless/scalastyle, cargo fmt -- all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
schenksj
force-pushed
the
pr/delta-A2-buildgate
branch
from
f2ad00c to
e020fe6
Compare
schenksj
added a commit
that referenced
this pull request
Jun 22, 2026
…flaky re-run) + apache#4366 carveout links
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this part is
Build gate + inert wiring. Establishes the
contrib-deltaMaven profile / Cargo feature and the inert wiring so a gated build compiles end to end, while the DEFAULT build stays byte-for-byte behavior-unchanged and carries zero Delta surface. No real Delta read logic yet — a Delta read that reaches native returns a cleanNotImplementedand falls back to vanilla Spark.Build gate
contrib-deltaprofile with per-Sparkdelta.version(3.5→3.3.2, 4.0→4.0.0, 4.1→4.1.0) + add-source ofcontrib/delta/src. Defaultspark.versionstays 4.1.2 (the delta-spark 4.1.1 pin is a separate, deferred decision).contrib-deltafeature (optional path dep on the contrib crate);native/Cargo.tomlexcludes../contribfrom the workspace.dev/verify-contrib-delta-gate.shproves default cargo/mvn/dylib carry zero Delta surface and the gated build pulls the right deps; wired into a minimaldelta_build_gate.ymljob.Inert wiring
Delta*messages +delta_scan = 118(117 isBroadcastNestedLoopJoin).OpStruct::DeltaScandispatch arm (not-compiled-in error on default builds; feature-gated shim into the contrib), exhaustive-match arms,convert_spark_types_to_arrow_schema→pub(crate).contrib/delta/native):plan_delta_scanreturnsDataFusionError::NotImplemented— just enough to satisfy the core shim so--features contrib-deltalinks.DeltaIntegration(reflective, returnsNoneuntil the contrib classes exist), the CometExecRule Delta-marker hook (CDF hook deferred), the CometScanRule Delta delegation + metadata-col reorder, and the leafDeltaConf.Why it's safe on default builds
No
-Pcontrib-delta/ no--features contrib-delta: the dispatch arm is a not-compiled-in error, the contrib crate isn't linked, andDeltaIntegration's reflective lookups resolve to nothing. The gate-verify script asserts this (0 Delta symbols in the defaultlibcomet, zeroio.deltain the effective pom, only theDeltaIntegrationbridge class compiled).Verification
default + gated native build, clippy (both feature states), the gate-verify script, gated + default JVM compile, spotless/scalastyle, and
cargo fmt— all green.🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.