[HIVE-4891] Distinct includes duplicate records

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 0.10.0
• Fix Version/s: 0.12.0
• Component/s: File Formats, HiveServer2, Query Processor
• Assignee: Unassigned
• Reporter: Fengdong Yu
• Created: 19/Jul/13 08:50
• Updated: 17/Oct/13 22:01
• Resolved: 25/Sep/13 07:14

Description

I have two partitions, one is sequence file, another is RCFile, but they are the same data(only different file format).

I have the following SQL:

select distinct uid from test where (dt ='20130718' or dt ='20130718_1') and cur_url like '%cq.aa.com%';

dt ='20130718' is sequence file,(default input format, which specified when create table)

dt ='20130718_1' is RCFile.

ALTER TABLE test ADD IF NOT EXISTS PARTITION (dt='20130718_1') LOCATION '/user/test/test-data'
ALTER TABLE test PARTITION(dt='20130718_1') SET FILEFORMAT RCFILE;

but there are duplicate recoreds in the result.

If two partitions with the same input format, then there are no duplicate records.