[TEZ-2186] OOM with a simple scatter gather job with re-use

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Target Version/s: 0.7.0, 0.5.4, 0.6.1
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Siddharth Seth
• Created: 09/Mar/15 20:32
• Updated: 30/Jun/15 04:53
• Resolved: 23/Mar/15 05:18

Description

With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found.