Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13127

"--yarnship" doesn't support resource classloading

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Release Note:
      This change orders the shipped files wrt to resources and jar files as well as alphabetically in these groups.

      Description

      Currently yarnship works as follows:

      • user specifies directory to ship with the job
      • yarn ships it with the container
      • org.apache.flink.yarn.AbstractYarnClusterDescriptor#uploadAndRegisterFiles traverses directory recursively and adds each file to the classpath

      This works well for shipping jars, but doesn't work correctly with shipping resources that we want to load using java.lang.ClassLoader#getResource method.

      In order to make resource classloading work, we need to register it's directory instead of the file itself (java classpath expects directories or archives).

      CLASSPATH="shipped/custom.conf:${CLASSPATH}" needs to become CLASSPATH="shipped:${CLASSPATH}"

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              dmvk David Morávek
              Reporter:
              dmvk David Morávek

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                  Issue deployment