Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15743

Prevent saving with all-column partitioning

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: SQL
    • Labels:

      Description

      When saving datasets on storage, `partitionBy` provides an easy way to construct the directory structure. However, if a user choose all columns as partition columns, some exceptions occurs.

      • ORC: `AnalysisException` on *future read* due to schema inference failure.
      • Parquet: `InvalidSchemaException` on *write execution* due to Parquet limitation.

      The followings are the examples.
      *ORC with all column partitioning*

      scala> spark.range(10).write.format("orc").mode("overwrite").partitionBy("id").save("/tmp/data")
                                                                                      
      scala> spark.read.format("orc").load("/tmp/data").collect()
      org.apache.spark.sql.AnalysisException: Unable to infer schema for ORC at /tmp/data. It must be specified manually;
      

      *Parquet with all-column partitioning*

      scala> spark.range(100).write.format("parquet").mode("overwrite").partitionBy("id").save("/tmp/data")
      [Stage 0:>                                                          (0 + 8) / 8]16/06/02 16:51:17 ERROR Utils: Aborting task
      org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not support empty group without leaves. Empty group: spark_schema
      ... (lots of error messages)
      

      Although some formats like JSON support all-column partitioning without any problem, it seems not a good idea to make lots of empty directories.

      This issue prevents this by consistently raising `AnalysisException` before saving.

        Attachments

          Activity

            People

            • Assignee:
              dongjoon Dongjoon Hyun
              Reporter:
              dongjoon Dongjoon Hyun
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: