[HIVE-21778] CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.5, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: CBO
• Labels: pull-request-available
• Assignee: Vineet Garg
• Reporter: Rajesh Balamohan
• Created: 22/May/19 12:16
• Updated: 27/Feb/24 22:23
• Resolved: 13/Mar/20 16:14

Description

drop table if exists test_struct;
CREATE external TABLE test_struct
(
  f1 string,
  demo_struct struct<f1:string, f2:string, f3:string>,
  datestr string
);

set hive.cbo.enable=true;
explain select * from etltmp.test_struct where datestr='2019-01-01' and demo_struct is not null;



STAGE PLANS:
  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        TableScan
          alias: test_struct
          filterExpr: (datestr = '2019-01-01') (type: boolean) <----- Note that demo_struct filter is not added here
          Filter Operator
            predicate: (datestr = '2019-01-01') (type: boolean)
            Select Operator
              expressions: f1 (type: string), demo_struct (type: struct<f1:string,f2:string,f3:string>), '2019-01-01' (type: string)
              outputColumnNames: _col0, _col1, _col2
              ListSink


set hive.cbo.enable=false;
explain select * from etltmp.test_struct where datestr='2019-01-01' and demo_struct is not null;


STAGE PLANS:
  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        TableScan
          alias: test_struct
          filterExpr: ((datestr = '2019-01-01') and demo_struct is not null) (type: boolean) <----- Note that demo_struct filter is added when CBO is turned off
          Filter Operator
            predicate: ((datestr = '2019-01-01') and demo_struct is not null) (type: boolean)
            Select Operator
              expressions: f1 (type: string), demo_struct (type: struct<f1:string,f2:string,f3:string>), '2019-01-01' (type: string)
              outputColumnNames: _col0, _col1, _col2
              ListSink

In CalcitePlanner::genFilterRelNode, the following code misses to evaluate this filter.

RexNode factoredFilterExpr = RexUtil
          .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);

Note that even if we add `demo_struct.f1` it would end up pushing the filter correctly.

Issue Links

• links to: GitHub Pull Request #928