[FLINK-18527] Task Managers fail to start on HDP 2.6.5 due to commons-cli conflict

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 1.11.0
• Component/s: Build System
• Labels: usability
• Assignee: Robert Metzger
• Reporter: Truong Duc Kien
• Created: 08/Jul/20 10:40
• Updated: 21/Sep/20 09:07
• Resolved: 21/Sep/20 09:00

Description

When launching a new job in HDP 2.6.5, we are encountering these exceptions

 

2020-07-08 16:10:36 E [default-dispatcher-4] [   o.a.f.y.YarnResourceManager] Could not start TaskManager in container container_xxx
java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
      at org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.<clinit>(CommandLineOptions.java:28) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
....
2020-07-08 16:12:46 E [default-dispatcher-4] [   o.a.f.y.YarnResourceManager] Could not start TaskManager in container container_xxx {} [] 
java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.runtime.entrypoint.parser.CommandLineOptions

We figure this is because HDP 2.6.5 is putting commons-cli version 1.2 on the class path, while Flink is expecting version 1.3. Maybe commons-cli should also be shaded to avoid such issue.