[SPARK-17267] Long running structured streaming requirements

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: Structured Streaming
• Target Version/s: 2.1.0
• Assignee: Peter Lee
• Reporter: Reynold Xin
• Created: 26/Aug/16 23:05
• Updated: 02/Nov/16 06:00
• Resolved: 02/Nov/16 05:59

Description

This is an umbrella ticket to track various things that are required in order to have the engine for structured streaming run non-stop in production.

Sub-Tasks

• MetadataLog should support purging old logs: Resolved
• FileStreamSource should not track the list of seen files indefinitely: Resolved
• Ability to remove old metadata for structure streaming MetadataLog: Resolved
• Change Source API so that sources do not need to keep unbounded state: Resolved
• HDFSMetadataLog should not leak CRC files: Resolved
• StreamExecution should discard unneeded metadata: Resolved