[SPARK-3916] recognize appended data in textFileStream()

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: DStreams
• Labels: bulk-closed
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 12/Oct/14 05:48
• Updated: 21/May/19 05:37
• Resolved: 21/May/19 05:37

Description

Right now, we only find new data from new files, the data written to old files (processed in last batch) will not be processed.

In order to support this, we need partialRDD(), which is an RDD for part of file.

Issue Links

• links to: [Github] Pull Request #2806 (davies)