[SPARK-16430] Add an option in file stream source to read 1 file at a time

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Structured Streaming
• Target Version/s: 2.0.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 07/Jul/16 20:45
• Updated: 19/Jun/17 17:54
• Resolved: 08/Jul/16 06:21

Description

An option that limits the file stream source to read 1 file at a time enables rate limiting. It has the additional convenience that a static set of files can be used like a stream for testing as this will allows those files to be considered one at a time.

Issue Links

• links to: [Github] Pull Request #14094 (tdas)
• links to: [Github] Pull Request #14143 (tdas)