Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30034 Use Apache Hive 2.3 dependency by default
  3. SPARK-30755

Update migration guide and add actionable exception for HIVE-15167

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      2020-01-27 05:11:20.446 - stderr> 20/01/27 05:11:20 INFO DAGScheduler: ResultStage 2 (main at NativeMethodAccessorImpl.java:0) failed in 1.000 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13, 10.110.21.210, executor 1): java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe
        2020-01-27 05:11:20.446 - stderr>  at java.lang.ClassLoader.defineClass1(Native Method)
        2020-01-27 05:11:20.446 - stderr>  at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
        2020-01-27 05:11:20.446 - stderr>  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        2020-01-27 05:11:20.446 - stderr>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
        2020-01-27 05:11:20.446 - stderr>  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        2020-01-27 05:11:20.446 - stderr>  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        2020-01-27 05:11:20.446 - stderr>  at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        2020-01-27 05:11:20.446 - stderr>  at java.security.AccessController.doPrivileged(Native Method)
        2020-01-27 05:11:20.446 - stderr>  at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        2020-01-27 05:11:20.446 - stderr>  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        2020-01-27 05:11:20.446 - stderr>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        2020-01-27 05:11:20.446 - stderr>  at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        2020-01-27 05:11:20.446 - stderr>  at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        2020-01-27 05:11:20.446 - stderr>  at java.lang.Class.forName0(Native Method)
        2020-01-27 05:11:20.446 - stderr>  at java.lang.Class.forName(Class.java:348)
        2020-01-27 05:11:20.446 - stderr>  at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:76)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:119)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:104)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:111)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:267)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:208)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.scheduler.Task.doRunTask(Task.scala:144)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.scheduler.Task.run(Task.scala:117)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$6(Executor.scala:567)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1559)
        2020-01-27 05:11:20.447 - stderr>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:570)
        2020-01-27 05:11:20.447 - stderr>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        2020-01-27 05:11:20.447 - stderr>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        2020-01-27 05:11:20.447 - stderr>  at java.lang.Thread.run(Thread.java:748)
        2020-01-27 05:11:20.447 - stderr> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.serde2.SerDe
        2020-01-27 05:11:20.447 - stderr>  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        2020-01-27 05:11:20.447 - stderr>  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        2020-01-27 05:11:20.447 - stderr>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        2020-01-27 05:11:20.447 - stderr>  at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        2020-01-27 05:11:20.447 - stderr>  ... 31 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yumwang Yuming Wang
                Reporter:
                yumwang Yuming Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: