What is the problem the feature request solves?
With spark.comet.parquet.write.enabled=true, a straightforward Parquet write still runs on Spark when the output path is an object store. CometDataWritingCommand.getSupportLevel rejects anything that isn't file: or hdfs::
if (!outputPath.startsWith("file:") && !outputPath.startsWith("hdfs:"))
return Unsupported("Supported output filesystems: local, HDFS")
Most production Spark writes target object stores (s3a://, gs://, abfss://), so native write is effectively unavailable there today.
Steps to reproduce
-- comet enabled, spark.comet.parquet.write.enabled=true, incompatible ops allowed
INSERT OVERWRITE DIRECTORY 's3a://example-bucket/warehouse/out'
USING parquet OPTIONS (compression 'zstd')
SELECT id, name FROM src; -- src is a native Comet scan
Explain shows Execute InsertIntoHadoopFsRelationCommand staying on Spark; extended Comet info reports the write operator as unsupported/disabled.
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
With
spark.comet.parquet.write.enabled=true, a straightforward Parquet write still runs on Spark when the output path is an object store.CometDataWritingCommand.getSupportLevelrejects anything that isn'tfile:orhdfs::Most production Spark writes target object stores (
s3a://,gs://,abfss://), so native write is effectively unavailable there today.Steps to reproduce
Explain shows
Execute InsertIntoHadoopFsRelationCommandstaying on Spark; extended Comet info reports the write operator as unsupported/disabled.Describe the potential solution
No response
Additional context
No response