Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6765

ASTNodeOrigin unserializable leads to fail when join with view

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.12.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      when a view contains a UDF, and the view comes into a JOIN operation, Hive will encounter a bug with stack trace like
      Caused by: java.lang.InstantiationException: org.apache.hadoop.hive.ql.parse.ASTNodeOrigin
      at java.lang.Class.newInstance0(Class.java:359)
      at java.lang.Class.newInstance(Class.java:327)
      at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)

        Activity

        Hide
        adrian-wang Adrian Wang added a comment -

        I added a PersistenceDelegate in serializeObject() in Class Utilities and resolved the problem. later I'll attach the patch.

        Show
        adrian-wang Adrian Wang added a comment - I added a PersistenceDelegate in serializeObject() in Class Utilities and resolved the problem. later I'll attach the patch.
        Hide
        adrian-wang Adrian Wang added a comment -

        Here's an example to see the Exception:
        CREATE TABLE t1 (a1 INT, b1 INT);
        CREATE VIEW v1 (x1) AS SELECT MAX(a1) FROM t1;
        SELECT s1.x1 FROM v1 s1 JOIN (SELECT MAX(a1) AS ma FROM t1) s2 ON s1.x1 = s2.ma;

        This is a bug on both ApacheHive and Tez, outputing "return code 1 ..."

        Show
        adrian-wang Adrian Wang added a comment - Here's an example to see the Exception: CREATE TABLE t1 (a1 INT, b1 INT); CREATE VIEW v1 (x1) AS SELECT MAX(a1) FROM t1; SELECT s1.x1 FROM v1 s1 JOIN (SELECT MAX(a1) AS ma FROM t1) s2 ON s1.x1 = s2.ma; This is a bug on both ApacheHive and Tez, outputing "return code 1 ..."
        Hide
        adrian-wang Adrian Wang added a comment -

        And I think this is just another drawback for using XMLEncoder to clone plan.

        Show
        adrian-wang Adrian Wang added a comment - And I think this is just another drawback for using XMLEncoder to clone plan.
        Hide
        adrian-wang Adrian Wang added a comment -

        Sorry, the previous example works on Tez with hive-0.13.
        But it fails when I run the query in Hive-0.12 in eclipse.

        Show
        adrian-wang Adrian Wang added a comment - Sorry, the previous example works on Tez with hive-0.13. But it fails when I run the query in Hive-0.12 in eclipse.
        Hide
        adrian-wang Adrian Wang added a comment -

        when I run the test case in hive command line(0.12-release), the full output is as follows,
        hive> SELECT s1.x1 FROM v1 s1 JOIN (SELECT MAX(a1) AS ma FROM t1) s2 ON s1.x1 = s2.ma;
        java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:361)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:277)
        at org.apache.hadoop.hive.ql.exec.Utilities.serializeObject(Utilities.java:666)
        at org.apache.hadoop.hive.ql.exec.Utilities.clonePlan(Utilities.java:637)
        at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:505)
        at org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
        at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
        at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
        at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
        at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
        at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:90)
        at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.compile(MapReduceCompiler.java:300)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8410)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
        Caused by: java.lang.Exception: XMLEncoder: discarding statement XMLEncoder.writeObject(MapredWork);
        ... 29 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeObject1(Encoder.java:231)
        at java.beans.Encoder.cloneStatement(Encoder.java:244)
        at java.beans.Encoder.writeStatement(Encoder.java:275)
        at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:348)
        ... 28 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264)
        ... 41 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:652)
        at java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:206)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:406)
        at java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:206)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:406)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264)
        ... 49 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:643)
        ... 65 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264)
        ... 77 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:652)
        at java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:206)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:406)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264)
        ... 90 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267)
        at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:643)
        ... 99 more
        Caused by: java.lang.RuntimeException: Cannot serialize object
        at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.Encoder.getValue(Encoder.java:108)
        at java.beans.Encoder.get(Encoder.java:225)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:110)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113)
        at java.beans.Encoder.writeObject(Encoder.java:74)
        at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274)
        at java.beans.Encoder.writeExpression(Encoder.java:304)
        at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389)
        at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229)
        at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264)
        ... 111 more
        Caused by: java.lang.InstantiationException: org.apache.hadoop.hive.ql.parse.ASTNodeOrigin
        at java.lang.Class.newInstance0(Class.java:359)
        at java.lang.Class.newInstance(Class.java:327)
        at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:55)
        at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262)
        at java.beans.Statement.invokeInternal(Statement.java:257)
        at java.beans.Statement.access$000(Statement.java:56)
        at java.beans.Statement$2.run(Statement.java:158)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.beans.Statement.invoke(Statement.java:154)
        at java.beans.Expression.getValue(Expression.java:115)
        at java.beans.Encoder.getValue(Encoder.java:105)
        ... 124 more
        FAILED: SemanticException Generate Map Join Task Error: Cannot serialize object

        Show
        adrian-wang Adrian Wang added a comment - when I run the test case in hive command line(0.12-release), the full output is as follows, hive> SELECT s1.x1 FROM v1 s1 JOIN (SELECT MAX(a1) AS ma FROM t1) s2 ON s1.x1 = s2.ma; java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:361) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:277) at org.apache.hadoop.hive.ql.exec.Utilities.serializeObject(Utilities.java:666) at org.apache.hadoop.hive.ql.exec.Utilities.clonePlan(Utilities.java:637) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:505) at org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79) at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:90) at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.compile(MapReduceCompiler.java:300) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8410) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.Exception: XMLEncoder: discarding statement XMLEncoder.writeObject(MapredWork); ... 29 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeObject1(Encoder.java:231) at java.beans.Encoder.cloneStatement(Encoder.java:244) at java.beans.Encoder.writeStatement(Encoder.java:275) at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:348) ... 28 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264) ... 41 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:652) at java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:206) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:406) at java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:206) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:406) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264) ... 49 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:643) ... 65 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264) ... 77 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:652) at java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:206) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:406) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264) ... 90 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:267) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:408) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:116) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:643) ... 99 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.Encoder.getValue(Encoder.java:108) at java.beans.Encoder.get(Encoder.java:225) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:110) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:113) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:274) at java.beans.Encoder.writeExpression(Encoder.java:304) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:389) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:229) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:264) ... 111 more Caused by: java.lang.InstantiationException: org.apache.hadoop.hive.ql.parse.ASTNodeOrigin at java.lang.Class.newInstance0(Class.java:359) at java.lang.Class.newInstance(Class.java:327) at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:55) at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262) at java.beans.Statement.invokeInternal(Statement.java:257) at java.beans.Statement.access$000(Statement.java:56) at java.beans.Statement$2.run(Statement.java:158) at java.security.AccessController.doPrivileged(Native Method) at java.beans.Statement.invoke(Statement.java:154) at java.beans.Expression.getValue(Expression.java:115) at java.beans.Encoder.getValue(Encoder.java:105) ... 124 more FAILED: SemanticException Generate Map Join Task Error: Cannot serialize object
        Hide
        selinazh Selina Zhang added a comment -

        Adrian Wang We came across the same problem. Just wonder why this ASTNodeOrigin serialization problem came up suddenly. It is not the first day we join with a view. The root cause maybe something else? Thanks!

        Show
        selinazh Selina Zhang added a comment - Adrian Wang We came across the same problem. Just wonder why this ASTNodeOrigin serialization problem came up suddenly. It is not the first day we join with a view. The root cause maybe something else? Thanks!
        Hide
        adrian-wang Adrian Wang added a comment -

        Selina Zhang Thanks for your comment! It's so glad that someone also noticed this. Actually, I found that only when there was something like an aggregation function in the view, will the problem came up. The problem results from cloning the plan, but when joining with view as described, the plan would contain a node of ASTNodeOrigin, which does not have a default construct method, in which case when duplicating, exception will be thrown.
        Could you please try to apply my patch here to see whether your problem is resolved? Thanks again.

        Show
        adrian-wang Adrian Wang added a comment - Selina Zhang Thanks for your comment! It's so glad that someone also noticed this. Actually, I found that only when there was something like an aggregation function in the view, will the problem came up. The problem results from cloning the plan, but when joining with view as described, the plan would contain a node of ASTNodeOrigin, which does not have a default construct method, in which case when duplicating, exception will be thrown. Could you please try to apply my patch here to see whether your problem is resolved? Thanks again.
        Hide
        cdrome Chris Drome added a comment -

        Adrian Wang this does not appear to be an issue with hive-0.13 as they are using Kryo for XML serialization.

        For clarification, I found that several setting can impact whether this problem arises or not. hive.auto.convert.join.noconditionaltask.size is used to determine the big table candidate set. Then hive.mapjoin.smalltable.filesize is used as a cutoff to determine whether a map-size join should be performed.

        I found that when hive.auto.convert.join is true, it will try to perform a map-side join first. Based on hive.auto.convert.join.noconditionaltask.size it will return a set of big table candidates to the physical optimizer. The physical optimizer will use hive.mapjoin.smalltable.filesize to determine whether the map-side join should proceed. If not the clonePlan method is called, which manifests the problem. I don't think it is solely influenced by the size of the tables involved in the join. In my tests, shrinking the size of the table allows the map-side join to proceed, while increasing the size of the table causes this failure.

        Show
        cdrome Chris Drome added a comment - Adrian Wang this does not appear to be an issue with hive-0.13 as they are using Kryo for XML serialization. For clarification, I found that several setting can impact whether this problem arises or not. hive.auto.convert.join.noconditionaltask.size is used to determine the big table candidate set. Then hive.mapjoin.smalltable.filesize is used as a cutoff to determine whether a map-size join should be performed. I found that when hive.auto.convert.join is true, it will try to perform a map-side join first. Based on hive.auto.convert.join.noconditionaltask.size it will return a set of big table candidates to the physical optimizer. The physical optimizer will use hive.mapjoin.smalltable.filesize to determine whether the map-side join should proceed. If not the clonePlan method is called, which manifests the problem. I don't think it is solely influenced by the size of the tables involved in the join. In my tests, shrinking the size of the table allows the map-side join to proceed, while increasing the size of the table causes this failure.
        Hide
        adrian-wang Adrian Wang added a comment -

        Chris Drome good catch. I knew the serialization in Hive has been notorious for a long time, but I didn't know the progress they made there. Actually, I was real curious when I saw my case was OK with Tez with hive-0.13, while I never tried Apache's hive-0.13 since there was no official release.

        Show
        adrian-wang Adrian Wang added a comment - Chris Drome good catch. I knew the serialization in Hive has been notorious for a long time, but I didn't know the progress they made there. Actually, I was real curious when I saw my case was OK with Tez with hive-0.13, while I never tried Apache's hive-0.13 since there was no official release.
        Hide
        ashutoshc Ashutosh Chauhan added a comment -

        Resolving as wont fix since 0.13 has been released which uses Kryo based serialization which doesnt have this problem.

        Show
        ashutoshc Ashutosh Chauhan added a comment - Resolving as wont fix since 0.13 has been released which uses Kryo based serialization which doesnt have this problem.

          People

          • Assignee:
            Unassigned
            Reporter:
            adrian-wang Adrian Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development