[TEZ-3350] Shuffle spills are not spilled to a container-specific directory

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.1
• Target Version/s: 0.7.2
• Assignee: László Bodor
• Reporter: Jason Darrell Lowe
• Created: 14/Jul/16 20:37
• Updated: 21/Feb/20 13:39

Description

If a Tez task receives too much input data and needs to spill the inputs to disk it ends up using a path that is not container-specific. Therefore YARN will not automatically cleanup these files when the container exits as it should, and instead the files linger until the entire application completes.

Issue Links

• is related to: TEZ-3877 Delete unordered spill files once merge is done