[SPARK-56125][SQL] Refactor MERGE INTO schema evolution to use assignment-based approach

[szehon-ho]

What changes were proposed in this pull request?

Refactors MERGE INTO schema evolution to compute pending schema changes directly from assignments instead of building an intermediate pruned source schema.

Before: pendingSchemaChanges called sourceSchemaForSchemaEvolution to build a pruned version of the source schema containing only columns referenced in assignments, then compared this pruned schema against the full target schema via computeSupportedSchemaChanges.

After: pendingSchemaChanges iterates over individual assignments that are schema evolution candidates and computes changes per-assignment:

• If the assignment key is unresolved (column missing from target), emit an addColumn change.
• If the key and value types differ, recursively compute schema changes on the types (handles nested struct additions and type widening).
• If types match, no change is needed.

Additionally:

• Extracted filterSupportedChanges from computeSupportedSchemaChanges in ResolveSchemaEvolution so MERGE can reuse the filtering logic independently.
• Made computeSchemaChanges private[catalyst] so it can be called from MergeIntoTable.
• Replaced originalTarget/originalSource parameters threaded through the recursive schema comparison with an onError callback, since these schemas were only used to construct the error message.
• Removed sourceSchemaForSchemaEvolution and its helpers (isEqual, isPrefix, isSameColumnAssignment).

Why are the changes needed?

The previous approach was indirect: it first constructed a pruned source schema, then compared it against the target schema to find differences. The new approach is more direct and easier to follow - it examines each assignment individually to determine what schema changes are needed. This also simplifies ResolveSchemaEvolution by removing the originalTarget/originalSource parameters that were only used for error reporting.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests cover MERGE INTO schema evolution behavior. This is a refactor with no behavioral change.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor (Claude Opus 4)

[szehon-ho]

this is from this suggestion : #55124 (comment) from @aokolnychyi

[szehon-ho]

FYI @aokolnychyi , applied offline review comments

[aokolnychyi]

I reconsider this suggestion. Let's remove this helper and instead add this one:

// a new helper method
private[catalyst] def computeSchemaChanges(
    targetType: StructType,
    sourceType: StructType,
    isByName: Boolean): Seq[TableChange] = {
  computeSchemaChanges(
    targetType,
    sourceType,
    fieldPath = Nil,
    isByName,
    throw QueryExecutionErrors.failedToMergeIncompatibleSchemasError(targetType, sourceType))
}

Then use it as the following:

def computeSupportedSchemaChanges(
    targetTable: LogicalPlan,
    sourceSchema: StructType,
    isByName: Boolean): Seq[TableChange] = {
  val candidateChanges = computeSchemaChanges(
    targetTable.schema,
    sourceSchema,
    isByName)
  filterSupportedChanges(targetTable, candidateChanges)
}

[aokolnychyi]

Keep throwIncompatibleSchemasError helper in MergeIntoTable but call QueryExecutionErrors.

[aokolnychyi]

Why remove the comments and docs?

[aokolnychyi]

Is this impacted by fix in #55340?

[szehon-ho]

i think not, that was changing the schemaEvolutionReady gate (prior to this logic)

This logic (schemaEvolutionTriggeringAssignments) is to decide what schema to modify

[aokolnychyi]

This got dropped in the master branch?

[aokolnychyi]

Same question here. Seems like this is accidental?