[HIVE-24551] Hive should include transitive dependencies from calcite after shading it

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.8
• Fix Version/s: 2.3.8
• Component/s: SQL
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 17/Dec/20 02:23
• Updated: 21/Dec/20 18:19
• Resolved: 21/Dec/20 18:19

Description

Currently as part of effort of shading Guava from Hive, we shade Calcite and exclude all its artifacts from the binary distribution. However, this also removes all its transitive dependencies which will still needed at runtime. Without these, Hive queries will fail with error like:

Exception in thread "main" java.lang.NoClassDefFoundError: org/eigenbase/util/property/BooleanProperty
        at org.apache.calcite.util.SaffronProperties.<init>(SaffronProperties.java:66)
        at org.apache.calcite.util.SaffronProperties.instance(SaffronProperties.java:134)
        at org.apache.calcite.util.Util.getDefaultCharset(Util.java:769)
        at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.getDefaultCharset(RelDataTypeFactoryImpl.java:565)
        at org.apache.calcite.sql.type.SqlTypeUtil.addCharsetAndCollation(SqlTypeUtil.java:1070)
        at org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:65)
        at org.apache.calcite.rex.RexBuilder.<init>(RexBuilder.java:114)
        at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:991)
        at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
        at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.ClassNotFoundException: org.eigenbase.util.property.BooleanProperty
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 33 more

This happens in branch-2.3 but the same thing might happen in master as well - although it uses calcite 1.21 which has changed a lot from 1.10 that branch-2.3 is using. For instance, in calcite 1.21 it no longer depends on org.eigenbase

Issue Links

• relates to: HIVE-23980 Shade guava from existing Hive versions
• links to: GitHub Pull Request #1792