[TEZ-4233] Map task should be blamed earlier for local fetch failures

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 12/Sep/20 14:15
• Updated: 29/Oct/23 18:19
• Resolved: 28/Sep/20 06:38

Description

Fetch failures can be a result of network issue or disk issue. Currently, AM doesn't know about whether the original input read error happened because of a local fetch failure or not. I think if a map output was reported as a subject of local fetch failure, AM should respond earlier, and blame it as soon as possible. Here is a hidden assumption that a disk read should never fail (or relatively rarely compared to network issues).

When I detected this issue, it was a Kubernetes based LLAP environment, where a daemon completely disappeared and a new daemon - running reducer tasks - assumed that it has map outputs locally, which wasn't the case.
This patch can help in container mode as well, as we can assume that a local read should work, and if it's not, the original map output data should be re-generated as soon as possible.

Issue Links

• is duplicated by: TEZ-4400 Tez takes a long time to recover from shuffle data not found errors