[BEAM-7854] Reading files from local file system does not fully support glob

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.16.0
• Component/s: sdk-java-core
• Labels: Done
• Assignee: Tomer Zeltzer
• Reporter: Tomer Zeltzer
• Created: 30/Jul/19 22:11
• Updated: 06/Oct/20 21:53
• Resolved: 13/Aug/19 14:50

Description

Folder structure:   

A
    B
        a=100
            data1
                file1.zst
                file2.zst 
        a=999 
            data2
                file6.zst
        a=397
            data3
                file7.zst

 

Glob:

 

/A/B/a=[0-9][0-9][0-9]/*/*

Code:  

 

input.apply(Create.of(patterns))
     .apply("Matching patterns", FileIO.matchAll())
     .apply(FileIO.readMatches());

 

input is of type PBegin.

The above code matches 0 files even though, from the glob, its clear it should match all files. I suspect its because of line 227, where only the first parent folder is checked while is could be an asterix in a glob. I believe the right behaviour should be to check all parent folder and use the first one that exists.

Issue Links

• causes: BEAM-8568 Local file system does not match relative path with wildcards
• links to: GitHub Pull Request #9197