[YARN-6356] Allow different values of yarn.log-aggregation.retain-seconds for succeeded and failed jobs

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: log-aggregation
• Assignee: Adam Antal
• Reporter: Robert Kanter
• Created: 16/Mar/17 21:41
• Updated: 05/Dec/19 18:35

Description

It would be useful to have a value of yarn.log-aggregation.retain-seconds for succeeded jobs and a different value for failed/killed jobs. For jobs that succeeded, you typically don't care about the logs, so a shorter retention time is fine (and saves space/blocks in HDFS). For jobs that failed or were killed, the logs are much more important, and it's likely to want to keep them around for longer so you have time to look at them.

For instance, you could set it to keep logs for succeeded jobs for 1 day and logs for failed/killed jobs for 1 week.