[TEZ-1731] OnDiskMerger can end up clobbering files across tasks with LocalDiskFetch

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.5.2
• Target Version/s: 0.5.2
• Assignee: Siddharth Seth
• Reporter: Siddharth Seth
• Created: 03/Nov/14 00:55
• Updated: 08/Nov/14 20:25
• Resolved: 03/Nov/14 21:41

Description

When an on disk fetch starts with LOCAL files (optimize.local.fetch), the filename used by the merger is based on the source file name. This name can be the same for all tasks reading the same input on the node - and can result in files being overwritten between tasks, depending on the order in which events are processed, and the dir allocated by the local dir-allocator.

Leads to ChecksumExceptions, and FileNotFoundExceptions during the merge.

Issue Links

• incorporates: TEZ-1727 FileName generated by OnDiskMerger is longer than required