Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13127

"--yarnship" doesn't support resource classloading

    XMLWordPrintableJSON

    Details

    • Release Note:
      This change orders the shipped files wrt to resources and jar files as well as alphabetically in these groups.

      Description

      Currently yarnship works as follows:

      • user specifies directory to ship with the job
      • yarn ships it with the container
      • org.apache.flink.yarn.AbstractYarnClusterDescriptor#uploadAndRegisterFiles traverses directory recursively and adds each file to the classpath

      This works well for shipping jars, but doesn't work correctly with shipping resources that we want to load using java.lang.ClassLoader#getResource method.

      In order to make resource classloading work, we need to register it's directory instead of the file itself (java classpath expects directories or archives).

      CLASSPATH="shipped/custom.conf:${CLASSPATH}" needs to become CLASSPATH="shipped:${CLASSPATH}"

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dmvk David Morávek
                Reporter:
                dmvk David Morávek
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m