[FLINK-14382] Incorrect handling of FLINK_PLUGINS_DIR on Yarn

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.9.0, 1.10.0
• Fix Version/s: 1.9.2, 1.10.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Yang Wang
• Reporter: Yang Wang
• Created: 12/Oct/19 12:35
• Updated: 27/Nov/19 08:35
• Resolved: 20/Nov/19 14:07

Description

(This ticket is a blocker for 1.10 release while critical for 1.9.x)

When creating and starting up the yarn containers there are two issues with how the FLINK_PLUGINS_DIR is being handled.

1. Content of the plugins directory is currently added to the class path, braking the encapsulation of the plugins from one another
2. FLINK_PLUGINS_DIR is passed to the container as an absolute path as seen by the client. Because of that TaskManager or JobManager can not use it.

Both bugs are probably contained to YarnClusterDescriptor#startAppMaster method (which calls relevant addEnvironmentFoldersToShipFiles and uploadAndRegisterFiles methods)

(original description)

If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to flink configuration according to the environment.

In yarn mode, the local path will be set in flink-conf.yaml and used by jobmanager and taskmanager. We will find the warning log like below.

2019-10-12 19:24:58,165 WARN  org.apache.flink.core.plugin.PluginConfig                     - Environment variable [FLINK_PLUGINS_DIR] is set to [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins] but the directory doesn't exist

It was in introduced by FLINK-12143.

Issue Links

• causes: FLINK-14968 Kerberized YARN on Docker test (custom fs plugin) fails on Travis
• relates to: FLINK-14561 Don't write FLINK_PLUGINS_DIR ENV variable to Flink configuration
• links to: GitHub Pull Request #10084
• links to: GitHub Pull Request #10211