[FLINK-20505] Yarn provided lib does not work with http paths.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 1.11.2, 1.12.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: zoucao
• Reporter: Xintong Song
• Created: 07/Dec/20 03:49
• Updated: 28/May/21 06:58
• Resolved: 30/Dec/20 03:34

Description

If an http path is used for provided lib, the following exception will be thrown on the resource manager side:

2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager - Could not start TaskManager in container containerXXXXXX.
org.apache.flink.util.FlinkException: Error to parse YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=XXXXX.jar, path=https://XXXXXXX.jar, size=-1, modificationTime=0, visibility=APPLICATION}
    at org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99)
    at org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721)
    at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626)
    at org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746)
    at org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726)
    at org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500)
    at org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455)
    at org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415)

The problem is that, `HttpFileSystem#getFilsStatus` returns file status with length `-1`, while `YarnLocalResourceDescriptor` does not recognize the negative file length.

Issue Links

• is related to: FLINK-20811 Support HTTP paths for yarn ship files/archives
• links to: GitHub Pull Request #14479