[TEZ-4338] Tez should consider node information to realize OUTPUT_LOST as early as possible - upstream(mapper) problems

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1
• Fix Version/s: 0.10.2
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 01/Oct/21 09:58
• Updated: 26/Dec/21 21:55
• Resolved: 19/Nov/21 10:11

Description

Finally, I decided to split TEZ-4139 into 2 different tasks, because handling upstream problems can be fixed independently and I'm focusing on that now

So, from TEZ-4139, this ticket is intended to handle downstream failures as:
collect all reported upstream mapper task attempts for a vertex, and if it's beyond a certain amount for the same source(map) host, blame mapper task immediately => blame mapper task attempt as soon as possible if read error is likely because of upstream node failure (somewhat similar goal to TEZ-3910, but currently TEZ-3910 is about to give the power of failure downstream task completely to AM)

Issue Links

• causes: TEZ-4364 TestFaultTolerance timeout on master - TestInput fix after TEZ-4338
• is related to: TEZ-4139 Tez should consider node information for computing failure fraction - downstream(reducer) problems
• links to: GitHub Pull Request #152