[FLINK-32809] YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work as expected

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: junzhong qin
• Reporter: junzhong qin
• Created: 08/Aug/23 11:13
• Updated: 11/Aug/23 04:52
• Resolved: 11/Aug/23 04:52

Description

YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List<File> shipFiles) check wether the shipFiles are all archive files, but it dose not work as expected.

public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List<File> shipFiles) {
    return shipFiles.stream()
            .filter(File::isFile)
            .map(File::getName)
            .map(String::toLowerCase)
            .allMatch(
                    name ->
                            name.endsWith(".tar.gz")
                                    || name.endsWith(".tar")
                                    || name.endsWith(".tgz")
                                    || name.endsWith(".dst")
                                    || name.endsWith(".jar")
                                    || name.endsWith(".zip"));
} 

When we pass a directory and an archive file it should return false but it returns true. 

// dir1 is a directory and archive.zip is an archive file
List<File> files = Arrays.asList(new File("/tmp/dir1"), new File("/tmp/archive.zip")); 
boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
System.out.println(result); // Print true but is should print false

If flink user want to add directory as ship file, they should use configuration: yarn.ship-files .

Issue Links

• links to: GitHub Pull Request #23191