Handle nulls in type coercion of higher-order UDFs, map_extract, spark repeat#23071
Open
gstvg wants to merge 1 commit into
Open
Handle nulls in type coercion of higher-order UDFs, map_extract, spark repeat#23071gstvg wants to merge 1 commit into
gstvg wants to merge 1 commit into
Conversation
…tract type coercion
github-actions
Bot
added
core
This was referenced Jun 22, 2026
| fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> { | ||
| let [map_type, _] = take_function_args(self.name(), arg_types)?; | ||
|
|
||
| if map_type.is_null() { |
Contributor
There was a problem hiding this comment.
Null is accepted in coercion, but return_type still calls get_map_entry_field on the same Null type. So
map_extract(NULL, 'a') still fails with an internal error.
We should also add a test for this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Unresolved lambda variables and unspecified placeholders report a
DataType::Nulldata type, which causes an error when used as argument of a function that doesn't handle nulls during type coercionWhat changes are included in this PR?
Handle null in
coerce_single_list_argfor higher-order functions,map_extractand in sparkrepeat. All other functions (scalar, window, agg and higher-order) that uses user defined type coercion have been checked to currently handle null args.Are these changes tested?
Test passing
array_filterwith an unresolved lambda variable toDataframe::with_column(which indirectly callsvalue_fields_with_higher_order_udf_and_lambdascoercion viaProjection::try_new,projection_schema,exprlist_to_fields,Expr::to_field)Are there any user-facing changes?
No