[SPARK-6061] File source dstream can not include the old file which timestamp is before the system time

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.2.1
• Component/s: DStreams
• Labels: FileSourceDStream OlderFiles Streaming
• Assignee: Unassigned
• Reporter: Jack Hu
• Created: 27/Feb/15 10:19
• Updated: 04/Apr/15 15:41
• Resolved: 04/Apr/15 15:41

Description

The file source dstream (StreamContext.fileStream) has a properties named "newFilesOnly" to include the old files, it worked fine with 1.1.0, and broken at 1.2.1, the older files always be ignored no mattern what value is set.

Here is the simple reproduce code:
https://gist.github.com/jhu-chang/1ee5b0788c7479414eeb

The reason is that: the "modTimeIgnoreThreshold" in FileInputDStream::findNewFiles is set to a time closed to system time (Spark Streaming Clock time), so the files old than this time are ignored.

Issue Links

• duplicates: SPARK-3276 Provide a API to specify MIN_REMEMBER_DURATION for files to consider as input in streaming