Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17624

MapredLocakTask running in separate JVM could throw ClassNotFoundException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.1
    • 3.0.0
    • Query Planning
    • None

    Description

      set hive.auto.convert.join=true;
      set hive.auto.convert.join.use.nonstaged=false;
      
      add jar hive-hcatalog-core.jar;
      
      drop table if exists t1;
      CREATE TABLE t1 (a string, b string)
      ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
      
      LOAD DATA LOCAL INPATH "data/files/sample.json" INTO TABLE t1;
      select * from t1 l join t1 r on l.a=r.a;
      

      The join will use a MapJoin which uses MapredLocalTask in a separate JVM to load the table into a Hashmap. But hive doesn't pass added jar to the classpath in such JVM so the following exception is thrown.

      org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
              at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:72)
              at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:92)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:564)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:140)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:127)
              at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
              at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
              at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
              at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:270)
              at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:69)
              ... 15 more
      
              at org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:586)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:140)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:127)
              at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
              at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
              at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
              at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      

      Attachments

        1. HIVE-17624.1.patch
          29 kB
          Aihua Xu
        2. HIVE-17624.2.patch
          56 kB
          Aihua Xu

        Activity

          People

            aihuaxu Aihua Xu
            aihuaxu Aihua Xu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: