[FLINK-13127] "--yarnship" doesn't support resource classloading

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.1
• Fix Version/s: 1.10.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Release Note: This change orders the shipped files wrt to resources and jar files as well as alphabetically in these groups.
• Assignee: David Morávek
• Reporter: David Morávek
• Created: 05/Jul/19 18:12
• Updated: 03/Dec/19 00:53
• Resolved: 15/Aug/19 17:59

Description

Currently yarnship works as follows:

• user specifies directory to ship with the job
• yarn ships it with the container
• org.apache.flink.yarn.AbstractYarnClusterDescriptor#uploadAndRegisterFiles traverses directory recursively and adds each file to the classpath

This works well for shipping jars, but doesn't work correctly with shipping resources that we want to load using java.lang.ClassLoader#getResource method.

In order to make resource classloading work, we need to register it's directory instead of the file itself (java classpath expects directories or archives).

CLASSPATH="shipped/custom.conf:${CLASSPATH}" needs to become CLASSPATH="shipped:${CLASSPATH}"

Issue Links

• is duplicated by: FLINK-6949 Add ability to ship custom resource files to YARN cluster
• links to: GitHub Pull Request #9022