[SPARK-20448] Document how FileInputDStream works with object storage

• Type: Documentation
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.3.0
• Component/s: Documentation
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 24/Apr/17 12:23
• Updated: 24/Sep/17 14:49
• Resolved: 23/Sep/17 16:08

Description

Object stores work differently from filesystems: intermediate writes not visible, renames are really O(data) copies, not O(1) transactions.

This makes working with them as DStreams fundamentally different: you can write straight into the destination.

1. Document how FileinputDStreams scan directories for changes
2. Document how object stores behave differently, and the implications
for users.

Issue Links

• relates to: SPARK-17159 Improve FileInputDStream.findNewFiles list performance
• links to: [Github] Pull Request #17743 (steveloughran)