[FLINK-10904] Expose Classloader before Pipeline execution

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Runtime / Task
• Assignee: Unassigned
• Reporter: Luka Jurukovski
• Created: 15/Nov/18 17:58
• Updated: 12/Apr/21 10:10

Description

Not sure if this is something that I just have to deal with. However it would be nice if the classloader can be accessed before the pipeline starts executing. The case for this is that I am loading classes that contain Flink operators. I am running into classdef not found issues because the classloader used by Flink is different then the application program that is being run. Currently what I have been doing as a work around is adding the libs that cause these issues in the /lib directory of the Flink cluster and marking it as provided in the application jar that is uploaded to the Flink cluster. The issues with this are two fold, first it makes deployment more complex, as well as there are cases where Classloading causes exceptions to arise in unusual circumstances. Ie RabbitMQ connector caused issues only when it was auto-recovering the connection, but not during normal ingest. I can elaborate further if needed