[HIVE-12788] Setting hive.optimize.union.remove to TRUE will break UNION ALL with aggregate functions

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.1
• Fix Version/s: 2.0.0
• Component/s: Hive
• Assignee: Chaoyu Tang
• Reporter: Eric Lin
• Created: 06/Jan/16 11:19
• Updated: 02/Jun/16 10:17
• Resolved: 13/Jan/16 16:09

Description

See the test case below:

0: jdbc:hive2://localhost:10000/default> create table test (a int);

0: jdbc:hive2://localhost:10000/default> insert overwrite table test values (1);

0: jdbc:hive2://localhost:10000/default> set hive.optimize.union.remove=true;
No rows affected (0.01 seconds)

0: jdbc:hive2://localhost:10000/default> set hive.mapred.supports.subdirectories=true;
No rows affected (0.007 seconds)

0: jdbc:hive2://localhost:10000/default> SELECT COUNT(1) FROM test UNION ALL SELECT COUNT(1) FROM test;
+----------+--+
| _u1._c0  |
+----------+--+
+----------+--+

UNION ALL without COUNT function will work as expected:

0: jdbc:hive2://localhost:10000/default> select * from test UNION ALL SELECT * FROM test;
+--------+--+
| _u1.a  |
+--------+--+
| 1      |
| 1      |
+--------+--+

Run the same query without setting hive.mapred.supports.subdirectories and hive.optimize.union.remove to true will give correct result:

0: jdbc:hive2://localhost:10000/default> set hive.optimize.union.remove;
+-----------------------------------+--+
|                set                |
+-----------------------------------+--+
| hive.optimize.union.remove=false  |
+-----------------------------------+--+

0: jdbc:hive2://localhost:10000/default> SELECT COUNT(1) FROM test UNION ALL SELECT COUNT(1) FROM test;
+----------+--+
| _u1._c0  |
+----------+--+
| 1        |
| 1        |
+----------+--+

Issue Links

• is depended upon by: HIVE-12840 stats optimize subqueries whenever possible in a union query