[SPARK-28594] Allow event logs for running streaming apps to be rolled over

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Labels: releasenotes
• Assignee: Jungtaek Lim
• Reporter: Stephen Levett
• Shepherd: Felix Cheung
• Created: 01/Aug/19 14:39
• Updated: 07/May/22 18:17
• Resolved: 29/Jan/20 23:12

Description

At all current Spark releases when event logging on spark streaming is enabled the event logs grow massively.  The files continue to grow until the application is stopped or killed.

The Spark history server then has difficulty processing the files.

https://issues.apache.org/jira/browse/SPARK-8617

Addresses .inprogress files but not event log files that are still running.

Identify a mechanism to set a "max file" size so that the file is rolled over when it reaches this size?

 

 

Issue Links

• links to: Design Doc