(This ticket is a blocker for 1.10 release while critical for 1.9.x)
When creating and starting up the yarn containers there are two issues with how the FLINK_PLUGINS_DIR is being handled.
- Content of the plugins directory is currently added to the class path, braking the encapsulation of the plugins from one another
- FLINK_PLUGINS_DIR is passed to the container as an absolute path as seen by the client. Because of that TaskManager or JobManager can not use it.
Both bugs are probably contained to YarnClusterDescriptor#startAppMaster method (which calls relevant addEnvironmentFoldersToShipFiles and uploadAndRegisterFiles methods)
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to flink configuration according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by jobmanager and taskmanager. We will find the warning log like below.
It was in introduced by