[HIVE-28218] AssertionError in VolcanoPlanner#fixUpInputs during MV rewritting

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: CBO
• Assignee: Stamatis Zampetakis Assign to me
• Reporter: Stamatis Zampetakis
• Created: 25/Apr/24 12:10
• Updated: 25/Apr/24 12:15

Description

The minimal scenario to reproduce the problem requires one ACID table, one JDBC table, and a view over the ACID table so that materialized view (MV) rewritting triggers. A minimal set of DDLs is outlined below.

CREATE TABLE emps_acid ( empid int, deptno int) STORED AS ORC TBLPROPERTIES ('transactional'='true');

CREATE MATERIALIZED VIEW mv_emps AS SELECT * FROM emps_acid;

CREATE EXTERNAL TABLE depts_jdbc (deptno int, name varchar(256)) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES (...);

EXPLAIN CBO SELECT * FROM emps_acid e JOIN depts_jdbc d;

When the MV rewritting is enabled the JOIN query above fails with the following stacktrace.

java.lang.AssertionError
	at org.apache.calcite.plan.volcano.VolcanoPlanner.fixUpInputs(VolcanoPlanner.java:1011)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1186)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:589)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:604)
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:148)
	at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
	at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
	at org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewBoxing$HiveMaterializedViewUnboxingRule.onMatch(HiveMaterializedViewBoxing.java:206)
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:229)
	at org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:58)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyMaterializedViewRewriting(CalcitePlanner.java:2053)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1715)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1572)
	at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
	at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1324)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:572)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13178)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:465)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
	at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
	at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
	at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)

A full reproducer is attached in materialized_view_rewrite_acid_jdbc_tables.q and can be run using:

mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_rewrite_acid_jdbc_tables.q -Dtest.output.overwrite