[HIVE-3890] Hive List Bucketing - merge per skewed dir

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Gang Tim Liu
• Reporter: Gang Tim Liu
• Created: 14/Jan/13 04:05
• Updated: 14/Jan/13 04:05

Description

Right now, in list bucketing DML, if it involves merge, it uses 1 MR job for all skewed directory. If no. of files is big, it might triggers hive client side OOM due to too many spits. If we use 1 MR job for one skewed dir, it will reduce OOM risks.