[SPARK-13791] Add MetadataLog and HDFSMetadataLog

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Structured Streaming
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 10/Mar/16 01:42
• Updated: 01/Nov/16 22:15
• Resolved: 15/Mar/16 02:29

Description

• Add a MetadataLog interface for metadata reliably storage.
• Add HDFSMetadataLog as a MetadataLog implementation based on HDFS.
• Update FileStreamSource to use HDFSMetadataLog instead of managing metadata by itself.

Issue Links

• links to: [Github] Pull Request #11625 (zsxwing)