[TEZ-4110] Make Tez fail fast when DFS quota is exceeded

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0, 0.8.4, 0.9.2
• Fix Version/s: 0.10.3

Environment

hadoop 2.9, hive 2.3, tez
 

• Assignee: Ayush Saxena
• Reporter: Wang Yan
• Created: 23/Dec/19 03:59
• Updated: 16/Oct/23 08:09
• Resolved: 16/Oct/23 07:50

Description

This ticket aims at creating a similar feature as MAPREDUCE-7148 in tez.

Make a tez job fail fast when dfs quota limitation is reached.

The background is : We are running hive jobs with a DFS quota limitation per job(3TB). If a job hits DFS quota limitation, the task that hit it will fail and there will be a few task reties before the job actually fails. The retry is not very helpful because the job will always fail anyway. In some worse cases, we have a job which has a single reduce task writing more than 3TB to HDFS over 20 hours, the reduce task exceeds the quota limitation and retries 4 times until the job fails in the end thus consuming a lot of unnecessary resource. This ticket aims at providing the feature to let a job fail fast when it writes too much data to the DFS and exceeds the DFS quota limitation.

 

Issue Links

• is related to: MAPREDUCE-7148 Fast fail jobs when exceeds dfs quota limitation
• is related to: HADOOP-16777 Add Tez to LimitedPrivate of ClusterStorageCapacityExceededException
• links to: GitHub Pull Request #313