[SPARK-7056] Make the WriteAheadLog pluggable

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.4.0
• Component/s: DStreams
• Target Version/s: 1.4.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 22/Apr/15 11:40
• Updated: 01/Aug/15 06:02
• Resolved: 29/Apr/15 20:06

Description

Users may want the WAL data to be written to non-HDFS data storage systems. To allow that, we have to make the WAL pluggable. The following design doc outlines the plan.

https://docs.google.com/a/databricks.com/document/d/1A2XaOLRFzvIZSi18i_luNw5Rmm9j2j4AigktXxIYxmY/edit?usp=sharing

Issue Links

• links to: [Github] Pull Request #5645 (tdas)