[HIVE-24080] CBO failed: Array and Map are not support when convert calcite types to hive types

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 2.3.4, 3.0.0
• Component/s: CBO, Logical Optimizer
• Assignee: Ashutosh Chauhan
• Reporter: Yizhen Fan
• Created: 27/Aug/20 09:15
• Updated: 28/Aug/20 02:52

Description

drop table temp.test_cbo_right;
CREATE EXTERNAL TABLE temp.test_cbo_right(
  `rid` string,
  `dtm` string,
  `map` map<string, string>,
  `tags` array <string>
);drop table temp.test_cbo_left;
CREATE EXTERNAL TABLE temp.test_cbo_left(
  `lid` string,
  `dtm` string
)

explain select
b.tags
from temp.test_cbo_left a
left join temp.test_cbo_right b on a.lid = b.rid
where b.dtm = '20200814'
and a.dtm = '20200813'

 

The Problem will occur in  left join and select array type or map type column in right table(reverse for right join or both for full join).

code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite and then convert it back to `optimizedAST`, the projection of b.tags has been optimized to be  RexCall => CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef, but in this case, calcite need cast `not null` to `nullable`.  However, TypeConverter in ASTConverter has not support ARRAY and MAP.

 

NPE threw here:

org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall

HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
if (ht.args != null) {
  for (String castArg : ht.args)
    astBldr.add(HiveParser.Identifier, castArg);
}

The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.

when table is large, sql will explode.

 

 I wonder which project should fix this problem