[HIVE-7767] hive.optimize.union.remove does not work properly [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Assignee: Na Yang
• Reporter: Na Yang
• Created: 18/Aug/14 17:34
• Updated: 29/May/15 02:32
• Resolved: 21/Aug/14 00:09

Description

Turing on the hive.optimize.union.remove property generates wrong union all result.

For Example:

create table inputTbl1(key string, val string) stored as textfile;
load data local inpath '../../data/files/T1.txt' into table inputTbl1;
SELECT *
FROM (
  SELECT key, count(1) as values from inputTbl1 group by key
  UNION ALL
  SELECT key, count(1) as values from inputTbl1 group by key
) a;  

when the hive.optimize.union.remove is turned on, the query result is like:

1	1
2	1
3	1
7	1
8	2

when the hive.optimize.union.remove is turned off, the query result is like:

7	1
2	1
8	2
3	1
1	1
7	1
2	1
8	2
3	1
1	1

The expected query result is:

7	1
2	1
8	2
3	1
1	1
7	1
2	1
8	2
3	1
1	1

Issue Links

• Is contained by: HIVE-7292 Hive on Spark
• is related to: HIVE-7541 Support union all on Spark [Spark Branch]
• is related to: HIVE-7717 Add .q tests coverage for "union all" [Spark Branch]