Describe the bug
TRY_CAST between map types whose key cast can produce null fails in Comet, where Spark returns
the row with a null key. Verified against main at 58ab5f618 and Spark 4.1.3.
Steps to reproduce
-- m is MAP<BIGINT, INT> holding a key that does not fit in INT
SELECT TRY_CAST(m AS MAP<INT, INT>) FROM t
Spark returns a map whose key is null. Comet fails the query.
Why Spark accepts the plan
Cast.canTryCast gates its map arm on !forceNullable(fromKey, toKey), and
forceNullable(LongType, IntegerType) matches none of the earlier arms, so it falls through to
case _ => false. LongType is an IntegralType, so the (_: FractionalType, _: IntegralType)
arm does not apply. A string key is rejected by case (_: StringType, _) => true, but a narrowing
integral key is not.
Spark then casts keys and values recursively through castArray and builds the result with
ArrayBasedMapData(keys, values). Nothing on that path rejects a null key, and under TRY the
inner cast turns the failure into a null.
Why it reaches Comet
CometCast.isSupported's map arm decides support purely by recursing into the key and value
casts:
case (from_map: MapType, to_map: MapType) =>
isSupported(from_map.keyType, to_map.keyType, timeZoneId, evalMode) match {
case Compatible(_, _) =>
isSupported(from_map.valueType, to_map.valueType, timeZoneId, evalMode)
case other => other
}
There is no equivalent of Spark's forceNullable gate, so the plan is marked Compatible and
runs natively.
What Comet does
cast_map_to_map builds the entries with the target field nullability. With a non-nullable target
key field and nulls among the cast keys, the construction fails. On main that is
StructArray::new, which is try_new(...).unwrap(), so it panics inside arrow and aborts the
executor. The branch in #5227 uses try_new and returns
Found unmasked nulls for non-nullable StructArray field "key"
so #5227 turns the panic into an error. Neither matches Spark.
Expected behavior
TRY_CAST should match Spark and produce a map with a null key.
Note this is specific to TRY_CAST. Plain CAST agrees with Spark in both modes: legacy wraps,
and ANSI raises [CAST_OVERFLOW].
Additional context
Found while narrowing #5227, which fixes field metadata and the target sorted flag in the same
function. Deliberately kept out of that PR, because this is a TRY_CAST semantics question
rather than a metadata preservation one.
Describe the bug
TRY_CASTbetween map types whose key cast can produce null fails in Comet, where Spark returnsthe row with a null key. Verified against
mainat58ab5f618and Spark 4.1.3.Steps to reproduce
Spark returns a map whose key is null. Comet fails the query.
Why Spark accepts the plan
Cast.canTryCastgates its map arm on!forceNullable(fromKey, toKey), andforceNullable(LongType, IntegerType)matches none of the earlier arms, so it falls through tocase _ => false.LongTypeis anIntegralType, so the(_: FractionalType, _: IntegralType)arm does not apply. A string key is rejected by
case (_: StringType, _) => true, but a narrowingintegral key is not.
Spark then casts keys and values recursively through
castArrayand builds the result withArrayBasedMapData(keys, values). Nothing on that path rejects a null key, and under TRY theinner cast turns the failure into a null.
Why it reaches Comet
CometCast.isSupported's map arm decides support purely by recursing into the key and valuecasts:
There is no equivalent of Spark's
forceNullablegate, so the plan is markedCompatibleandruns natively.
What Comet does
cast_map_to_mapbuilds the entries with the target field nullability. With a non-nullable targetkey field and nulls among the cast keys, the construction fails. On
mainthat isStructArray::new, which istry_new(...).unwrap(), so it panics inside arrow and aborts theexecutor. The branch in #5227 uses
try_newand returnsso #5227 turns the panic into an error. Neither matches Spark.
Expected behavior
TRY_CASTshould match Spark and produce a map with a null key.Note this is specific to
TRY_CAST. PlainCASTagrees with Spark in both modes: legacy wraps,and ANSI raises
[CAST_OVERFLOW].Additional context
Found while narrowing #5227, which fixes field metadata and the target sorted flag in the same
function. Deliberately kept out of that PR, because this is a
TRY_CASTsemantics questionrather than a metadata preservation one.