[HIVE-27939] Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 4.0.0-beta-1
• Component/s: Query Processor
• Labels: pull-request-available
• Assignee: Ryu Kobayashi
• Reporter: Ryu Kobayashi
• Created: 07/Dec/23 09:06
• Updated: 10/Jul/24 14:25
• Resolved: 07/Mar/24 08:33

Description

I found that the ticket for HIVE-26779 alone does not resolve when using many UNION ALL. When we create a DDL with ddl.sql and execute a query with query.sql, we get a SemanticException similar to HIVE-26779.

23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent
org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to remove partition predicates: fail to find child from parent
        at org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
        at org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
        at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
        at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
        at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
        at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

Issue Links

• relates to: HIVE-27858 OOM happens when selecting many columns and JOIN.
• links to: GitHub Pull Request #4950
• links to: GitHub Pull Request #4992