[HIVE-26458] Add explicit dependency to commons-dbcp2 in hive-exec module

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Labels: pull-request-available
• Target Version/s: 4.0.0-alpha-2
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 05/Aug/22 16:55
• Updated: 16/Nov/22 13:50
• Resolved: 15/Aug/22 13:48

Description

Hive CBO relies on Calcite so there is a direct dependency towards Calcite in hive-exec module. On its turn, Calcite needs commons-dbcp2 dependency in order to compile and run properly:

https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/build.gradle.kts#L69

In particular the dependency is necessary in order to use the JDBC adapter and some of its usages are shown below:

• https://github.com/apache/calcite/blob/257c81b5cac35e29598a246463356fea7e0b0336/core/src/main/java/org/apache/calcite/adapter/jdbc/JdbcUtils.java#L29
• https://github.com/apache/calcite/blob/257c81b5cac35e29598a246463356fea7e0b0336/core/src/main/java/org/apache/calcite/adapter/jdbc/JdbcUtils.java#L262

However, due to the shading of Calcite inside hive-exec module all the transitive dependencies coming from Calcite must be defined explicitly otherwise they will not make it to the classpath.

At the moment this does not pose a problem in master since commons-dbcp2 dependency comes transitively from other modules. But in certain Hive branches with slightly different dependencies between modules we have seen failures like the one shown below:

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/commons/dbcp2/BasicDataSource
	at org.apache.calcite.adapter.jdbc.JdbcUtils$DataSourcePool.<init>(JdbcUtils.java:213)
	at org.apache.calcite.adapter.jdbc.JdbcUtils$DataSourcePool.<clinit>(JdbcUtils.java:210)
	at org.apache.calcite.adapter.jdbc.JdbcSchema.dataSource(JdbcSchema.java:207)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3331)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5324)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1815)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1750)
	at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
	at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.plan(CalcitePlanner.java:1411)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:588)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13071)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:472)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:312)
	at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
	at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:650)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:596)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:590)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:231)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:421)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:352)
	at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:867)
	at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837)
	at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:178)
	at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:173)
	at org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:151)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:111)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/dbcp2/BasicDataSource
	... 71 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.dbcp2.BasicDataSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 71 more

Without shading there wouldn't be a need to have an explicit dependency to dbcp2 but the way it is currently done it is neccessary to have this. This is the same reason of why janino and other transitive deps from Calcite are declared explicitly in hive-exec module.

Issue Links

• is related to: HIVE-23346 Hive is Using o.a.c dbcp2 From hive-druid-handler JAR
• relates to: HIVE-22126 hive-exec packaging should shade guava
• links to: GitHub Pull Request #3510