Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20350

[Kinesis][GCP PubSub] Incompatible Connectors due to Guava conflict

    XMLWordPrintableJSON

    Details

      Description

      Problem
      Kinesis and GCP PubSub connector do not work together. The following error is thrown.

      java.lang.NoClassDefFoundError: Could not initialize class io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder
      	at org.apache.flink.streaming.connectors.gcp.pubsub.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:52) ~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.createAndSetPubSubSubscriber(PubSubSource.java:213) ~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.open(PubSubSource.java:102) ~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) ~[flink-core-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291) ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473) ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469) ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522) ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-runtime_2.11-1.11.1.jar:1.11.1]
      	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-runtime_2.11-1.11.1.jar:1.11.1]
      	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_252]
      
      <dependency>
          <groupId>org.apache.flink</groupId>
          <artifactId>flink-connector-gcp-pubsub_${scala.binary.version}</artifactId>
          <version>1.11.1</version>
      </dependency>
      
      <dependency>
         <groupId>org.apache.flink</groupId>
          <artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId>
          <version>1.11.1</version>
      </dependency>
      

      Cause
      This is caused by a Guava dependency conflict:

      • Kinesis Consumer > 18.0
      • GCP PubSub > 26.0-android

      NettyChannelBuilder fails to initialise due to missing method in guava:

      • com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V

      Possible Fixes

      • Align Guava versions
      • Shade Guava in either connector

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              danny.cranmer Danny Cranmer
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: