Hive
  1. Hive
  2. HIVE-5379

NoClassDefFoundError is thrown when using lead/lag with kryo serialization

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: PTF-Windowing
    • Labels:
      None

      Description

      java.lang.RuntimeException: Error in configuring object
      	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
      	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
      	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:432)
      	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
      	at org.apache.hadoop.mapred.Child.main(Child.java:260)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
      	... 9 more
      Caused by: java.lang.NoClassDefFoundError: org/antlr/runtime/tree/TreeWizard$ContextVisitor
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
      	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	at java.lang.Class.getDeclaringClass(Native Method)
      	at java.lang.Class.getEnclosingClass(Class.java:1085)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1054)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1110)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
      	at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:812)
      	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:723)
      	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:736)
      	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:304)
      	at org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:262)
      	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:115)
      	... 14 more
      Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.tree.TreeWizard$ContextVisitor
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      

      Seemed LeadLagInfo in PTFTranslator is the problem.

      1. D13155.1.patch
        7 kB
        Phabricator

        Activity

        Hide
        Eric Chu added a comment -

        Please ignore the above. False alarm. Sorry for the inconvenience.

        Show
        Eric Chu added a comment - Please ignore the above. False alarm. Sorry for the inconvenience.
        Hide
        Eric Chu added a comment -

        To continue from above, we applied this patch and the query above worked, but the query "SELECT sum(num) OVER (PARTITION BY name) FROM test_sum" would give the following exception:
        Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.parse.PTFTranslator$LeadLagInfo
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
        ...

        Show
        Eric Chu added a comment - To continue from above, we applied this patch and the query above worked, but the query "SELECT sum(num) OVER (PARTITION BY name) FROM test_sum" would give the following exception: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.parse.PTFTranslator$LeadLagInfo at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136) ...
        Hide
        Eric Chu added a comment -

        We had the same error when we ran:
        SELECT count(num) OVER (PARTITION BY name) FROM test_sum;

        Show
        Eric Chu added a comment - We had the same error when we ran: SELECT count(num) OVER (PARTITION BY name) FROM test_sum;
        Hide
        Hudson added a comment -

        ABORTED: Integrated in Hive-trunk-hadoop2 #461 (See https://builds.apache.org/job/Hive-trunk-hadoop2/461/)
        HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Show
        Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #461 (See https://builds.apache.org/job/Hive-trunk-hadoop2/461/ ) HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2363 (See https://builds.apache.org/job/Hive-trunk-h0.21/2363/)
        HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2363 (See https://builds.apache.org/job/Hive-trunk-h0.21/2363/ ) HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #183 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/183/)
        HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #183 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/183/ ) HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #117 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/117/)
        HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #117 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/117/ ) HIVE-5379 - NoClassDefFoundError is thrown when using lead/lag with kryo serialization (Reviewed By Ashutosh, Contributed by Navis) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526941 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
        Hide
        Brock Noland added a comment -

        Hey guys I committed this to trunk since I was committing another patch. Thank you for the Review Ashutosh and thank you for the patch Navis!

        Show
        Brock Noland added a comment - Hey guys I committed this to trunk since I was committing another patch. Thank you for the Review Ashutosh and thank you for the patch Navis!
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12605374/D13155.1.patch

        SUCCESS: +1 3176 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/932/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/932/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12605374/D13155.1.patch SUCCESS: +1 3176 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/932/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/932/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        Brock Noland added a comment -

        Nice find and thanks for the fix! +1 as well.

        Show
        Brock Noland added a comment - Nice find and thanks for the fix! +1 as well.
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-5379 [jira] NoClassDefFoundError is thrown when using lead/lag with kryo serialization".

        +1

        REVISION DETAIL
        https://reviews.facebook.net/D13155

        BRANCH
        HIVE-5379

        ARCANIST PROJECT
        hive

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-5379 [jira] NoClassDefFoundError is thrown when using lead/lag with kryo serialization". +1 REVISION DETAIL https://reviews.facebook.net/D13155 BRANCH HIVE-5379 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis
        Hide
        Navis added a comment -

        Just moving LLInfo out of PTFTranslator fixed the problem.

        Show
        Navis added a comment - Just moving LLInfo out of PTFTranslator fixed the problem.
        Hide
        Phabricator added a comment -

        navis requested code review of "HIVE-5379 [jira] NoClassDefFoundError is thrown when using lead/lag with kryo serialization".

        Reviewers: JIRA

        HIVE-5379 NoClassDefFoundError is thrown when using lead/lag with kryo serialization

        java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:432)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
        at org.apache.hadoop.mapred.Child.main(Child.java:260)
        Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 9 more
        Caused by: java.lang.NoClassDefFoundError: org/antlr/runtime/tree/TreeWizard$ContextVisitor
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.getDeclaringClass(Native Method)
        at java.lang.Class.getEnclosingClass(Class.java:1085)
        at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1054)
        at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1110)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502)
        at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
        at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
        at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:812)
        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:723)
        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:736)
        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:304)
        at org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:262)
        at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:115)
        ... 14 more
        Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.tree.TreeWizard$ContextVisitor
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

        Seemed LeadLagInfo in PTFTranslator is the problem.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D13155

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java

        MANAGE HERALD RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/38481/

        To: JIRA, navis

        Show
        Phabricator added a comment - navis requested code review of " HIVE-5379 [jira] NoClassDefFoundError is thrown when using lead/lag with kryo serialization". Reviewers: JIRA HIVE-5379 NoClassDefFoundError is thrown when using lead/lag with kryo serialization java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:432) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 9 more Caused by: java.lang.NoClassDefFoundError: org/antlr/runtime/tree/TreeWizard$ContextVisitor at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.Class.getDeclaringClass(Native Method) at java.lang.Class.getEnclosingClass(Class.java:1085) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1054) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1110) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:812) at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:723) at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:736) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:304) at org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:262) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:115) ... 14 more Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.tree.TreeWizard$ContextVisitor at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Seemed LeadLagInfo in PTFTranslator is the problem. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D13155 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/LeadLagInfo.java ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingExprNodeEvaluatorFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/38481/ To: JIRA, navis

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development