[HIVE-12462] DPP: DPP optimizers need to run on the TS predicate not FIL

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Invalid
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Tez
• Target Version/s: 2.0.0
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 19/Nov/15 06:24
• Updated: 16/Feb/16 23:53
• Resolved: 06/Jan/16 19:28

Description

HIVE-11398 + HIVE-11791, the partition-condition-remover became more effective.

This removes predicates from the FilterExpression which involve partition columns, causing a miss for dynamic-partition pruning if the DPP relies on FilterDesc.

The TS desc will have the correct predicate in that condition.

$hdt$_0:$hdt$_1:a
  TableScan (TS_2)
    alias: a
    filterExpr: (((account_id = 22) and year(dt) is not null) and (year(dt)) IN (RS[6])) (type: boolean)
    Filter Operator (FIL_20)
      predicate: ((account_id = 22) and year(dt) is not null) (type: boolean)
      Select Operator (SEL_4)
        expressions: dt (type: date)
        outputColumnNames: _col1
        Reduce Output Operator (RS_8)
          key expressions: year(_col1) (type: int)
          sort order: +
          Map-reduce partition columns: year(_col1) (type: int)
          Join Operator (JOIN_9)
            condition map:
                 Inner Join 0 to 1
            keys:
              0 year(_col1) (type: int)
              1 year(_col1) (type: int)

Issue Links

• is related to: HIVE-12666 PCRExprProcFactory.GenericFuncExprProcessor.process() aggressively removes dynamic partition pruner generated synthetic join predicates.