Description
When an on disk fetch starts with LOCAL files (optimize.local.fetch), the filename used by the merger is based on the source file name. This name can be the same for all tasks reading the same input on the node - and can result in files being overwritten between tasks, depending on the order in which events are processed, and the dir allocated by the local dir-allocator.
Leads to ChecksumExceptions, and FileNotFoundExceptions during the merge.
Attachments
Attachments
Issue Links
- incorporates
-
TEZ-1727 FileName generated by OnDiskMerger is longer than required
-
- Closed
-