Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7837

NPE when save as parquet in speculative tasks

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      The query is like df.orderBy(...).saveAsTable(...).

      When there is no partitioning columns and there is a skewed key, I found the following exception in speculative tasks. After these failures, seems we could not call SparkHadoopMapRedUtil.commitTask correctly.

      java.lang.NullPointerException
      	at parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:146)
      	at parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:112)
      	at parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:73)
      	at org.apache.spark.sql.parquet.ParquetOutputWriter.close(newParquet.scala:115)
      	at org.apache.spark.sql.sources.DefaultWriterContainer.abortTask(commands.scala:385)
      	at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.org$apache$spark$sql$sources$InsertIntoHadoopFsRelation$$writeRows$1(commands.scala:150)
      	at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:122)
      	at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:122)
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
      	at org.apache.spark.scheduler.Task.run(Task.scala:70)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lian cheng Cheng Lian
                Reporter:
                yhuai Yin Huai
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: