Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14382

Incorrect handling of FLINK_PLUGINS_DIR on Yarn

    XMLWordPrintableJSON

    Details

      Description

      (This ticket is a blocker for 1.10 release while critical for 1.9.x)

      When creating and starting up the yarn containers there are two issues with how the FLINK_PLUGINS_DIR is being handled.

      1. Content of the plugins directory is currently added to the class path, braking the encapsulation of the plugins from one another
      2. FLINK_PLUGINS_DIR is passed to the container as an absolute path as seen by the client. Because of that TaskManager or JobManager can not use it.

      Both bugs are probably contained to YarnClusterDescriptor#startAppMaster method (which calls relevant addEnvironmentFoldersToShipFiles and uploadAndRegisterFiles methods)

      (original description)

      If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to flink configuration according to the environment.

      In yarn mode, the local path will be set in flink-conf.yaml and used by jobmanager and taskmanager. We will find the warning log like below.

      2019-10-12 19:24:58,165 WARN  org.apache.flink.core.plugin.PluginConfig                     - Environment variable [FLINK_PLUGINS_DIR] is set to [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins] but the directory doesn't exist
      

      It was in introduced by FLINK-12143.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fly_in_gis Yang Wang
                Reporter:
                fly_in_gis Yang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m