[FLINK-3655] Allow comma-separated or multiple directories to be specified for FileInputFormat

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.5.0
• Labels: pull-request-available starter
• Assignee: Fabian Hueske
• Reporter: Gna Phetsarath
• Created: 22/Mar/16 18:41
• Updated: 09/Feb/21 00:44
• Resolved: 16/Feb/18 11:21

Description

Allow comma-separated or multiple directories to be specified for FileInputFormat so that a DataSource will process the directories sequentially.

env.readFile("/data/2016/01/01//,/data/2016/01/02//,/data/2016/01/03//")

in Scala

env.readFile(paths: Seq[String])
or
env.readFile(path: String, otherPaths: String*)

Wildcard support would be a bonus.

Issue Links

• relates to: FLINK-6417 Wildcard support for FileInputFormat
• relates to: FLINK-3677 FileInputFormat: Allow to specify include/exclude file name patterns
• links to: GitHub Pull Request #1990
• links to: GitHub Pull Request #5415
• links to: GitHub Pull Request #5415