[HIVE-13200] Aggregation functions returning empty rows on partitioned columns

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 2.0.0
• Fix Version/s: 1.3.0, 2.0.1, 2.1.0
• Component/s: Physical Optimizer
• Assignee: Yongzhi Chen
• Reporter: Yongzhi Chen
• Created: 03/Mar/16 16:29
• Updated: 21/Jun/16 15:43
• Resolved: 05/Mar/16 16:12

Description

Running aggregation functions like MAX, MIN, DISTINCT against partitioned columns will return empty rows if table has property: 'skip.header.line.count'='1'
Reproduce:

DROP TABLE IF EXISTS test;

CREATE TABLE test (a int) 
PARTITIONED BY (b int) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' 
TBLPROPERTIES('skip.header.line.count'='1');

INSERT OVERWRITE TABLE test PARTITION (b = 1) VALUES (1), (2), (3), (4);
INSERT OVERWRITE TABLE test PARTITION (b = 2) VALUES (1), (2), (3), (4);

SELECT * FROM test;

SELECT DISTINCT b FROM test;
SELECT MAX(b) FROM test;
SELECT DISTINCT a FROM test;

The output:

0: jdbc:hive2://localhost:10000/default> SELECT * FROM test;
+---------+---------+--+
| test.a  | test.b  |
+---------+---------+--+
| 2       | 1       |
| 3       | 1       |
| 4       | 1       |
| 2       | 2       |
| 3       | 2       |
| 4       | 2       |
+---------+---------+--+
6 rows selected (0.631 seconds)

0: jdbc:hive2://localhost:10000/default> SELECT DISTINCT b FROM test;
+----+--+
| b  |
+----+--+
+----+--+
No rows selected (47.229 seconds)

0: jdbc:hive2://localhost:10000/default> SELECT MAX(b) FROM test;
+-------+--+
|  _c0  |
+-------+--+
| NULL  |
+-------+--+
1 row selected (49.508 seconds)

0: jdbc:hive2://localhost:10000/default> SELECT DISTINCT a FROM test;
+----+--+
| a  |
+----+--+
| 2  |
| 3  |
| 4  |
+----+--+
3 rows selected (46.859 seconds)

Issue Links

• is related to: HIVE-12950 get rid of the NullScan emptyFile madness