[HIVE-10607] Combination of ReducesinkDedup + TopN optimization yields incorrect result if there are multiple GBY in reducer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0, 0.14.0, 1.0.0, 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Logical Optimizer, Tez
• Assignee: Ashutosh Chauhan
• Reporter: Ashutosh Chauhan
• Created: 05/May/15 00:41
• Updated: 18/May/15 19:49
• Resolved: 06/May/15 07:01

Description

select ctinyint, count(cdouble) from (select ctinyint, cdouble from alltypesorc group by ctinyint, cdouble) t1 group by ctinyint order by ctinyint limit 20;

This gives different result set depending on which set of optimizations are on. In particular in .q test environment following two invocations will give you different result set:

*   mvn test -Phadoop-2 -Dtest.output.overwrite=true -Dtest=TestMiniTezCliDriver -Dqfile=test.q -Dhive.optimize.reducededuplication.min.reducer=1 -Dhive.limit.pushdown.memory.usage=0.3f

*   mvn test -Phadoop-2 -Dtest.output.overwrite=true -Dtest=TestMiniTezCliDriver -Dqfile=test.q 

Issue Links

• blocks: HIVE-10568 Select count(distinct()) can have more optimal execution plan
• links to: RB request