[SPARK-15990] Support rolling log aggregation for Spark running on YARN

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 16/Jun/16 16:46
• Updated: 17/May/20 18:14
• Resolved: 29/Jun/16 13:18

Description

YARN supports rolling log aggregation since version 2.6+, it will aggregate the logs in a timely manner and upload to the HDFS. Compared to the previous log aggregation method which only aggregate the logs after application is finished, this will speed up the log aggregation time. Also this will avoid too large log file problem (out of disk).

So here propose to introduce this feature for Spark on YARN.

Issue Links

• links to: [Github] Pull Request #13712 (jerryshao)