[HIVE-20281] SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.2.0, 4.0.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: Physical Optimizer
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Ashutosh Chauhan
• Created: 31/Jul/18 16:16
• Updated: 27/Feb/24 22:23
• Resolved: 02/Aug/18 16:52

Description

HIVE-18201 seems to trigger a latent bug in SW optimizer. Test subquery_in_having fails with:

2018-07-31T08:42:57,328 DEBUG [b68f20cc-54d5-466d-b512-1540b3a43396 main] optimizer.SharedWorkOptimizer: After SharedWorkExtendedOptimizer:
TS[0]-SEL[1]-MAPJOIN[131]-FIL[12]-SEL[13]-GBY[14]-RS[15]-GBY[16]-SEL[17]-MAPJOIN[136]-MAPJOIN[137]-FIL[103]-SEL[104]-FS[105]
     -FIL[113]-SEL[20]-RS[44]-MAPJOIN[133]-SEL[47]-GBY[48]-RS[49]-GBY[50]-SEL[51]-GBY[55]-RS[98]-MAPJOIN[136]
                                                          -RS[88]-GBY[89]-SEL[120]-FIL[116]-SEL[91]-GBY[93]-RS[94]-GBY[95]-SEL[96]-RS[101]-MAPJOIN[137]
TS[2]-FIL[112]-GBY[5]-RS[6]-GBY[7]-SEL[8]-RS[10]-MAPJOIN[131]
                                         -RS[31]-MAPJOIN[132]-FIL[33]-SEL[34]-GBY[35]-RS[36]-GBY[37]-SEL[38]-GBY[42]-MAPJOIN[133]
TS[21]-FIL[114]-SEL[22]-MAPJOIN[132]
2018-07-31T08:42:57,329 ERROR [b68f20cc-54d5-466d-b512-1540b3a43396 main] ql.Driver: FAILED: SemanticException Error in shared work optimizer: operator cache contentsand actual plan differ
org.apache.hadoop.hive.ql.parse.SemanticException: Error in shared work optimizer: operator cache contentsand actual plan differ
        at org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:524)
        at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:185)
        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:146)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12361)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:663)

Issue Links

• relates to: HIVE-18201 Disable XPROD_EDGE for sq_count_check() created for scalar subqueries