[TEZ-2192] Relocalization does not check for source

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.6.0, 0.5.2
• Fix Version/s: 0.5.4, 0.6.1
• Target Version/s: 0.5.4, 0.6.1
• Assignee: Hitesh Shah
• Reporter: Rohini Palaniswamy
• Created: 10/Mar/15 23:32
• Updated: 30/Jun/15 04:52
• Resolved: 06/Apr/15 21:51

Description

PIG-4443 spills the input splits to disk if serialized split size is greater than some threshold. It faces issues with relocalization when more than one vertex has job.split file. If a job.split file is already there on container reuse, it is reused causing wrong data to be read.

Either need a way to turn off relocalization or check the source+timestamp and redownload the file during relocalization.

Issue Links

• breaks: PIG-4443 Write inputsplits in Tez to disk if the size is huge and option to compress pig input splits