[spark] Add load_csv and export_csv procedures

[JunRuiLee]

Purpose

In our scenario, many algorithm engineers work directly with datasets in CSV format. This PR adds Spark load_csv and export_csv procedures to make it easy to move data between CSV files and Paimon tables without writing custom Spark jobs.

load_csv imports CSV files into an existing Paimon table. It matches CSV header columns to target table columns by exact name, writes missing columns as null, drops extra columns, and always uses Spark CSV PERMISSIVE mode so malformed rows are counted in invalid_count and skipped. Nested columns are restored from JSON strings.

export_csv exports a Paimon table to a Spark CSV output directory, with optional where filtering. Nested columns are serialized as JSON strings, and quoteAll=true is enabled by default so JSON values containing commas are quoted correctly. Existing output paths are overwritten.

Tests

Added CsvProcedureTest.

[JingsongLi]

Cool!

[JingsongLi]

Maybe it is better to support COPY INTO? It seems that many products are designed in this way.

[JunRuiLee]

Maybe it is better to support COPY INTO? It seems that many products are designed in this way.

@JingsongLi Thanks for the suggestion.

After checking existing systems, I found two common directions:

1. Databricks-style import
   In Databricks, COPY INTO is mainly used for loading files into tables. Following this model, we can use COPY INTO only for importing files into Paimon tables, while keeping CSV export as a Spark procedure.

2. Snowflake-style bidirectional COPY
   In Snowflake, COPY INTO supports both loading data into tables and unloading data to files. Following this model, we would use COPY INTO for both import and export.

I prefer starting with Option 1: support Databricks-style COPY INTO for import first, with CSV as the first supported format, and keep export as a procedure. This is closer to the Spark ecosystem and keeps the initial scope smaller. Snowflake-style export can be discussed separately later if needed.

[JingsongLi]

Hi @JunRuiLee , I think we can try to look directly at Snowflake's perspective and see if there are any substantial bottlenecks.

[JunRuiLee]

Hi @JunRuiLee , I think we can try to look directly at Snowflake's perspective and see if there are any substantial bottlenecks.

Thanks @JingsongLi for suggestion, I'll take a look.

[JunRuiLee]

Closing this PR as its contents have been superseded by PR #7926