[HIVE-19101] Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge number of predicates

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
• Component/s: CBO
• Assignee: Unassigned
• Reporter: Ganesha Shreedhara
• Created: 04/Apr/18 07:24
• Updated: 27/Feb/24 22:24
• Resolved: 18/May/18 09:04

Description

Hive query is getting stuck during the optimisation phase while applying HiveJoinPushTransitivePredicatesRule when there are huge number of predicates.

 

DEBUG Log:

2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10963: Apply rule [ReduceExpressionsRule(Join)] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10964: Apply rule [HiveJoinAddNotNullRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10965: Apply rule [HiveJoinPushTransitivePredicatesRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]

 

Thread Status:

 

"6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800 nid=0x1c03 runnable [0x0000700008176000]
 java.lang.Thread.State: RUNNABLE
 at java.util.Arrays.copyOfRange(Arrays.java:2694)
 at java.lang.String.<init>(String.java:203)
 at java.lang.StringBuilder.toString(StringBuilder.java:405)
 at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
 at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
 at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
 at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
 at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
 at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
 at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
 at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
 at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
 at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
 at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
 at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
 at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
 at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
 at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
 at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
 at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
 at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
 at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
 at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
 at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
 at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
 at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
 at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
 at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
 at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
 at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Locked ownable synchronizers:
 - <0x00000007d5cfcab0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

 

 

This issue seems to be  because of optimisation done on CBO (Specifically on calcite transitive predicate inference rule) in hive 2.1.1 and later versions. 

This issue is not there in hive1.2.1 version.

 

The queries to reproduce this issue is in this attachment : queries. 

There is a view (test_view) created by combining the results from 16 tables using UNION ALL. 

The query is getting stuck when I run a select query which performs join between view (test_view) and a table (test1). 

 

Issue Links

• relates to: HIVE-19433 HiveJoinPushTransitivePredicatesRule hangs