[SPARK-17159] Improve FileInputDStream.findNewFiles list performance

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0
• Component/s: DStreams

Environment

spark against object stores

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/Aug/16 19:04
• Updated: 05/Oct/18 01:22
• Resolved: 05/Oct/18 01:22

Description

FileInputDStream.findNewFiles() is doing a globStatus with a fitler that calls getFileStatus() on every file, takes the output and does listStatus() on the output.

This going to suffer on object stores, as dir listing and getFileStatus calls are so expensive. It's clear this is a problem, as the method has code to detect timeouts in the window and warn of problems.

It should be possible to make this faster

Issue Links

• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: HADOOP-13946 Document how HDFS updates timestamps in the FS spec; compare with object stores
• is related to: SPARK-20448 Document how FileInputDStream works with object storage
• relates to: SPARK-7481 Add spark-hadoop-cloud module to pull in object store support
• links to: [Github] Pull Request #14731 (steveloughran)
• links to: [Github] Pull Request #17745 (steveloughran)
• links to: [Github] Pull Request #22339 (ScrapCodes)