[FLINK-14561] Don't write FLINK_PLUGINS_DIR ENV variable to Flink configuration

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.1, 1.10.0
• Fix Version/s: 1.9.2, 1.10.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 29/Oct/19 14:21
• Updated: 31/Oct/19 13:02
• Resolved: 31/Oct/19 13:02

Description

With FLINK-12143 we introduced the plugin mechanism. As part of this feature, we now write the FLINK_PLUGINS_DIR environment variable to the Flink Configuration we use for the cluster components. This is problematic, because we also use this Configuration to start new processes (Yarn and Mesos TaskExecutors). If the Configuration contains a configured FLINK_PLUGINS_DIR which differs from the one used by the newly created process, then this leads to problems.

In order to solve this problem, I suggest to not write env variables which are intended for local usage within the started process into the Configuration. Instead we should directly read the environment variable at the required site similar to what we do with the env variable FLINK_LIB_DIR.

Issue Links

• is caused by: FLINK-12143 Mechanism to ship plugin jars in the cluster
• is related to: FLINK-14382 Incorrect handling of FLINK_PLUGINS_DIR on Yarn
• links to: GitHub Pull Request #10037