[TEZ-3440] Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.2, 0.9.0, 0.8.5
• Assignee: Nathan Roberts
• Reporter: Nathan Roberts
• Created: 21/Sep/16 17:51
• Updated: 22/Jan/24 08:46
• Resolved: 05/Oct/16 15:32

Description

Haven't verified yet but certainly looks like tez needs same fix as MAPREDUCE-5308 in IFile.

Specifically saw this because downstream tasks were reporting enough fetch failures that long-running upstream tasks had to be re-run, which makes job run for much longer than it needs.

Usually shows itself as an "Invalid map id" error on a multi-map fetch on part 2-n (i.e. never the first one).

Issue Links

• is related to: TEZ-4533 Source release distribution must not contain TestIFile_concatenated_compressed.bin