Skip to content

Native Parquet write only supports file:/hdfs: outputs — falls back on s3a:// #6007

Description

@parthchandra

What is the problem the feature request solves?

With spark.comet.parquet.write.enabled=true, a straightforward Parquet write still runs on Spark when the output path is an object store. CometDataWritingCommand.getSupportLevel rejects anything that isn't file: or hdfs::

if (!outputPath.startsWith("file:") && !outputPath.startsWith("hdfs:"))
  return Unsupported("Supported output filesystems: local, HDFS")

Most production Spark writes target object stores (s3a://, gs://, abfss://), so native write is effectively unavailable there today.

Steps to reproduce

-- comet enabled, spark.comet.parquet.write.enabled=true, incompatible ops allowed
INSERT OVERWRITE DIRECTORY 's3a://example-bucket/warehouse/out'
USING parquet OPTIONS (compression 'zstd')
SELECT id, name FROM src;   -- src is a native Comet scan

Explain shows Execute InsertIntoHadoopFsRelationCommand staying on Spark; extended Comet info reports the write operator as unsupported/disabled.

Describe the potential solution

No response

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions