Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3365

Failure to save Lists to Parquet

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Reproduction, same works if type is Seq. (props to Chris Grier for finding this)

      scala> case class Test(x: List[String])
      defined class Test
      
      scala> sparkContext.parallelize(Test(List()) :: Nil).saveAsParquetFile("bug")
      23:09:51.807 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
      java.lang.ArithmeticException: / by zero
      	at parquet.hadoop.InternalParquetRecordWriter.initStore(InternalParquetRecordWriter.java:99)
      	at parquet.hadoop.InternalParquetRecordWriter.<init>(InternalParquetRecordWriter.java:92)
      	at parquet.hadoop.ParquetRecordWriter.<init>(ParquetRecordWriter.java:64)
      	at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:282)
      	at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:252)
      	at org.apache.spark.sql.parquet.InsertIntoParquetTable.org$apache$spark$sql$parquet$InsertIntoParquetTable$$writeShard$1(ParquetTableOperations.scala:300)
      	at org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
      	at org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
      	at org.apache.spark.scheduler.Task.run(Task.scala:54)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

        Attachments

          Activity

            People

            • Assignee:
              lian cheng Cheng Lian
              Reporter:
              marmbrus Michael Armbrust
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: