[FLINK-7927] Different Netty Versions in dependencies of flink-runtime make it impossible to use 3rd party libraries using netty

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.2
• Fix Version/s: 1.4.0
• Component/s: Build System

Environment

• Windows 10 x64
• Java 1.8

• Assignee: Unassigned
• Reporter: Claudius Eisele
• Created: 25/Oct/17 20:48
• Updated: 12/Jan/18 01:22
• Resolved: 12/Jan/18 01:22

Description

I am trying to use Google PubSub (google-cloud-pubsub 0.26.0-beta) in a Flink streaming job but I am receiving the following error when executing it so unfortunately it's not possible to use PubSub in a Flink Streaming Job:

...
10/25/2017 22:38:02     Source: Custom Source -> Map(1/1) switched to RUNNING
10/25/2017 22:38:03     Source: Custom Source -> Map(1/1) switched to FAILED
java.lang.IllegalStateException: Expected the service InnerService [FAILED] to be RUNNING, but the service has FAILED
        at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:328)
        at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:266)
        at com.google.api.core.AbstractApiService.awaitRunning(AbstractApiService.java:97)
....

Caused by: java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
        at io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:159)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:136)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:124)
        at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:94)
        at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(NettyChannelBuilder.java:525)
        at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(NettyChannelBuilder.java:518)
        at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:457)
        at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:326)
        at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:315)
        at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:131)
        at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:116)
        at com.google.cloud.pubsub.v1.Subscriber.doStart(Subscriber.java:246)
        at com.google.api.core.AbstractApiService$InnerService.doStart(AbstractApiService.java:149)
        at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:211)
        at com.google.api.core.AbstractApiService.startAsync(AbstractApiService.java:121)
        at com.google.cloud.pubsub.v1.Subscriber.startAsync(Subscriber.java:235)
        ... 7 more

I reported this problem to the Google Cloud Java Library but the problem seems more to be in Flink or its dependencies like akka because there are a lot of netty dependencies with different versions in it:

• Apache Zookeeper (flink-runtime dependency) has --- io.netty:netty:3.7.0.Final -> 3.8.0.Final
• Flakka (flink-runtime dependency) has io.netty:netty:3.8.0.Final
• Flink-Runtime has io.netty:netty-all:4.0.27.Final

In my case, Google Cloud PubSub has io.grpc:grpc-netty:1.6.1

Additional information on the issue in combination with Google Cloud PubSub can be found here:
https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2398
https://github.com/grpc/grpc-java/issues/3025