[PIG-4649] [Pig on Tez] Union followed by HCatStorer misses some data

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0, 0.15.1
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 04/Aug/15 17:58
• Updated: 08/Jun/16 20:48
• Resolved: 05/Aug/15 20:22

Description

Script to reproduce:

A = LOAD 'data01.txt' USING PigStorage() as (id:chararray, message:chararray);
B = LOAD 'data02.txt' USING PigStorage() as (id:chararray, message:chararray);
C = UNION A, B;
STORE C INTO 'db.table1' USING org.apache.hive.hcatalog.pig.HCatStorer();

Issue Links

• is broken by: HIVE-11456 HCatStorer should honor mapreduce.output.basename
• relates to: PIG-4694 MultiStorageOutputFormat does not honor mapreduce.base.outputname
• relates to: PIG-4691 [Pig on Tez] Support for whitelisting storefuncs for union optimization