[SPARK-15698] Ability to remove old metadata for structure streaming MetadataLog

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: Structured Streaming
• Target Version/s: 2.0.1, 2.1.0
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 01/Jun/16 09:27
• Updated: 28/Dec/16 18:15
• Resolved: 20/Sep/16 19:54

Description

Current MetadataLog lacks the ability to remove old checkpoint file, we'd better add this functionality to the MetadataLog and honor it in the place where MetadataLog is used, that will reduce unnecessary small files in the long running scenarios.

Issue Links

• is depended upon by: SPARK-17604 Support purging aged file entry for FileStreamSource metadata log
• links to: [Github] Pull Request #13513 (jerryshao)
• links to: [Github] Pull Request #15163 (zsxwing)
• links to: [Github] Pull Request #15173 (jerryshao)