[PIG-4691] [Pig on Tez] Support for whitelisting storefuncs for union optimization

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed

Release Note

Union optimization (pig.tez.opt.union=true) in tez uses vertex groups to store output from different vertices into one final output location. If a StoreFunc's OutputCommitter does not honor mapreduce.output.basename or has other issues with multiple vertices writing to the destination location at the same time, then you can disable union optimization just for that StoreFunc. Refer PIG-4649. You can also specify a whitelist of StoreFuncs that are known to work with multiple vertices writing to same location instead of a blacklist.

#pig.tez.opt.union.unsupported.storefuncs=org.apache.hcatalog.pig.HCatStorer,org.apache.hive.hcatalog.pig.HCatStorer
#pig.tez.opt.union.supported.storefuncs=

• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 05/Oct/15 22:40
• Updated: 08/Jun/16 20:48
• Resolved: 06/Oct/15 21:23

Description

PIG-4649 added support for blacklisting some storefuncs when applying union+store vertex group optimization as HCatStorer was not honoring mapreduce.output.basename and hardcoding part file names. Found that some of our user StoreFuncs also do that and ended up with partial results. So would be good to have a whitelist option as well where you can put StoreFuncs that do not mess with mapreduce.output.basename.

Issue Links

• is related to: PIG-4649 [Pig on Tez] Union followed by HCatStorer misses some data