[MAPREDUCE-7148] Fast fail jobs when exceeds dfs quota limitation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0, 2.8.0, 2.9.0
• Fix Version/s: 3.3.0
• Component/s: task

Environment

hadoop 2.7.3

• Target Version/s: 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Wang Yan
• Reporter: Wang Yan
• Created: 05/Oct/18 02:17
• Updated: 24/Dec/19 02:17
• Resolved: 07/Nov/18 14:31

Description

We are running hive jobs with a DFS quota limitation per job(3TB). If a job hits DFS quota limitation, the task that hit it will fail and there will be a few task reties before the job actually fails. The retry is not very helpful because the job will always fail anyway. In some worse cases, we have a job which has a single reduce task writing more than 3TB to HDFS over 20 hours, the reduce task exceeds the quota limitation and retries 4 times until the job fails in the end thus consuming a lot of unnecessary resource. This ticket aims at providing the feature to let a job fail fast when it writes too much data to the DFS and exceeds the DFS quota limitation. The fast fail feature is introduced in MAPREDUCE-7022 and MAPREDUCE-6489 .

Issue Links

• relates to: TEZ-4110 Make Tez fail fast when DFS quota is exceeded