Skip to content

TRY_CAST on narrowing map keys fails where Spark returns a map with a null key #5995

Description

@Smallfu666

Describe the bug

TRY_CAST between map types whose key cast can produce null fails in Comet, where Spark returns
the row with a null key. Verified against main at 58ab5f618 and Spark 4.1.3.

Steps to reproduce

-- m is MAP<BIGINT, INT> holding a key that does not fit in INT
SELECT TRY_CAST(m AS MAP<INT, INT>) FROM t

Spark returns a map whose key is null. Comet fails the query.

Why Spark accepts the plan

Cast.canTryCast gates its map arm on !forceNullable(fromKey, toKey), and
forceNullable(LongType, IntegerType) matches none of the earlier arms, so it falls through to
case _ => false. LongType is an IntegralType, so the (_: FractionalType, _: IntegralType)
arm does not apply. A string key is rejected by case (_: StringType, _) => true, but a narrowing
integral key is not.

Spark then casts keys and values recursively through castArray and builds the result with
ArrayBasedMapData(keys, values). Nothing on that path rejects a null key, and under TRY the
inner cast turns the failure into a null.

Why it reaches Comet

CometCast.isSupported's map arm decides support purely by recursing into the key and value
casts:

case (from_map: MapType, to_map: MapType) =>
  isSupported(from_map.keyType, to_map.keyType, timeZoneId, evalMode) match {
    case Compatible(_, _) =>
      isSupported(from_map.valueType, to_map.valueType, timeZoneId, evalMode)
    case other => other
  }

There is no equivalent of Spark's forceNullable gate, so the plan is marked Compatible and
runs natively.

What Comet does

cast_map_to_map builds the entries with the target field nullability. With a non-nullable target
key field and nulls among the cast keys, the construction fails. On main that is
StructArray::new, which is try_new(...).unwrap(), so it panics inside arrow and aborts the
executor. The branch in #5227 uses try_new and returns

Found unmasked nulls for non-nullable StructArray field "key"

so #5227 turns the panic into an error. Neither matches Spark.

Expected behavior

TRY_CAST should match Spark and produce a map with a null key.

Note this is specific to TRY_CAST. Plain CAST agrees with Spark in both modes: legacy wraps,
and ANSI raises [CAST_OVERFLOW].

Additional context

Found while narrowing #5227, which fixes field metadata and the target sorted flag in the same
function. Deliberately kept out of that PR, because this is a TRY_CAST semantics question
rather than a metadata preservation one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions