[HIVE-22363] ReduceDeduplication may leave an invalid GroupByOperator behind in some cases

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Physical Optimizer
• Labels: pull-request-available
• Target Version/s: 3.1.3, 3.2.0, 4.0.0
• Assignee: Zoltan Haindrich
• Reporter: Zoltan Haindrich
• Created: 17/Oct/19 15:09
• Updated: 07/Jun/20 00:26

Description

since HIVE-11387 reducededup may traverse GroupByOperators as well

But the removal logic only removes the first parent; so if there is some other operator (a FIL in this case) between the sink and the gby - the removal may not happen here

set hive.cbo.enable=false;

drop table if exists xl1;
create table xl1 as
select '1' as mdl_yr_desc, 2 as seq_no,'3' as opt_desc1,4 as opt_desc,1 as row_num;

explain
select trim(base.mdl_yr_desc) mdl_yr_desc, trim(base.opt_desc) opt_desc
from
(
    SELECT trim(mdl_yr_desc) mdl_yr_desc, concat_ws(' ', collect_set(trim(opt_desc1))) AS opt_desc
    from
    (
        select t14304.* 
        from
        (
            select * from xl1
        ) t14304  
        where row_num = 1
        order by trim(mdl_yr_desc), cast(seq_no as int) asc
    ) x
    group by trim(mdl_yr_desc)
) base
inner join
    (
        select 1 as v
    ) dedup
    on  trim(base.mdl_yr_desc) != dedup.v
group by trim(base.mdl_yr_desc), trim(base.opt_desc) ;

Issue Links

• links to: GitHub Pull Request #819