[SPARK-29779] Compact old event log files and clean up

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 06/Nov/19 21:25
• Updated: 31/Jan/20 23:09
• Resolved: 10/Jan/20 17:53

Description

This issue is to track the effort on compacting old event logs (and cleaning up after compaction) without breaking guaranteeing of compatibility.

Please note that this issue leaves below functionalities for future JIRA issue as the patch for SPARK-29779 is too huge and we decided to break down.

• apply filter in SQL events
• integrate compaction into FsHistoryProvider
• documentation about new configuration

Issue Links

• blocks: SPARK-30479 Apply compaction of event log to SQL events
• links to: GitHub Pull Request #26416
• links to: GitHub Pull Request #27085
• links to: GitHub Pull Request #27086
• links to: GitHub Pull Request #27087
• links to: GitHub Pull Request #27414