Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18022

java.lang.NullPointerException instead of real exception when saving DF to MySQL

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.2, 2.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Hi,
      I have found following issue.
      When there is an exception while saving dataframe to MySQL I'm unable to get it.
      Instead of I'm getting following stacktrace.

      16/10/20 06:00:35 WARN TaskSetManager: Lost task 56.0 in stage 10.0 (TID 3753, dwh-hn28.adpilot.co): java.lang.NullPointerException: Cannot suppress a null exception.
              at java.lang.Throwable.addSuppressed(Throwable.java:1046)
              at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:256)
              at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:314)
              at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:313)
              at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
              at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
              at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1916)
              at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1916)
              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
              at org.apache.spark.scheduler.Task.run(Task.scala:86)
              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

      The real exception could be for example duplicate on primary key etc.

      With this it's very difficult to debugging apps.

        Attachments

          Activity

            People

            • Assignee:
              srowen Sean Owen
              Reporter:
              maver1ck Maciej BryƄski
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: