[FLINK-14896] Kinesis connector doesn't shade jackson dependency

• Type: Bug
• Status: Closed
• Priority: Not a Priority
• Resolution: Fixed
• Affects Version/s: 1.9.0, 1.16.0, 1.15.2
• Fix Version/s: 1.17.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available

Environment

AWS EMR 5.28.0

• Release Note: Shade and relocate transitive Jackson dependencies of {{flink-connector-kinesis}}. If your Flink job was transitively relying on the these, you may need to include additional Jackson dependencies into your project.
• Assignee: Danny Cranmer
• Reporter: Michel Davit
• Created: 21/Nov/19 13:00
• Updated: 13/Jun/23 09:08
• Resolved: 27/Sep/22 07:38

Description

flink-kinesis-connector depends on aws java sdk which is shaded to org.apache.flink.kinesis.shaded.com.amazonaws.

 

However, the aws sdk has a transitive dependency to jackson wich is not shaded in the artifact.

 

This creates problem when running flink on YARN: The aws sdk requires jackson-core v2.6 but hadoop pulls in 2.3. See here.

 

If YARN uses the loads wrong jackson version from classpath. Jod fails with

2019-11-20 17:23:11,563 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler    - Unhandled exception.org.apache.flink.client.program.ProgramInvocationException: The program caused an error:     at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)    at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:80)    at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:126)    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$6(JarRunHandler.java:142)    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;    at com.amazonaws.partitions.PartitionsLoader.<clinit>(PartitionsLoader.java:54)    at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30)    at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:65)    at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:53)    at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:107)    at com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256)    at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:460)    at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)    at com.amazonaws.client.builder.AwsAsyncClientBuilder.build(AwsAsyncClientBuilder.java:80)
...

The flink-kinesis-connector should do as other connectors: shade jackson or use the flink-shaded-jackson core dependency

Issue Links

• links to: GitHub Pull Request #10285
• links to: GitHub Pull Request #20865