[HIVE-3734] Static partition DML create duplicate files and records

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.10.0
• Component/s: Query Processor
• Assignee: Gang Tim Liu Assign to me
• Reporter: Gang Tim Liu
• Created: 22/Nov/12 02:03
• Updated: 29/Nov/12 19:32
• Resolved: 27/Nov/12 21:57

Description

Static DML create duplicate files and record.

Given the following test case, hive will return 2 records:
484 val_484
484 val_484

but srcpart returns one record:
484 val_484

If you look at file system, DML generates duplicate file with the same content:
rw-rr- 1 gang THEFACEBOOK\Domain Users 5812 Nov 21 17:55 000000_0
-rwxr-xr-x 1 gang THEFACEBOOK\Domain Users 5812 Nov 21 17:55 000001_0

Test Case
===
set hive.mapred.supports.subdirectories=true;
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set hive.merge.mapfiles=false;
set hive.merge.mapredfiles=false;
set mapred.input.dir.recursive=true;

create table testtable (key String, value String) partitioned by (ds String, hr String) ;

explain extended
insert overwrite table testtable partition (ds='2008-04-08', hr='11') select key, value from srcpart where ds='2008-04-08';
insert overwrite table testtable partition (ds='2008-04-08', hr='11') select key, value from srcpart where ds='2008-04-08';

desc formatted testtable partition (ds='2008-04-08', hr='11');

select count(1) from srcpart where ds='2008-04-08';
select count(1) from testtable where ds='2008-04-08';

select key, value from srcpart where ds='2008-04-08' and hr='11' and key = "484";
explain extended
select key, value from testtable where ds='2008-04-08' and hr='11' and key = "484";
select key, value from testtable where ds='2008-04-08' and hr='11' and key = "484";
===

Issue Links

• relates to: HIVE-3073 Hive List Bucketing - DML support