[FLINK-19995] 【Flink SQL Client】Use Flink Kafka Connector in SQL-Client has more than one exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.12.0
• Component/s: Table SQL / Ecosystem
• Assignee: Unassigned
• Reporter: zhisheng
• Created: 05/Nov/20 09:48
• Updated: 22/Apr/21 01:42
• Resolved: 06/Nov/20 02:58

Description

when i add flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar in lib, I run  sql job has an exception like picture2

 

[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

when i add flink-connector-kafka_2.11-1.12-SNAPSHOT.jar in lib, it run has another exception

 

[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArrayDeserializer

if i add both jar, it returm exception too

[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArrayDeserializer

ddl & sql:

 

CREATE TABLE UserBehavior (
 user_id BIGINT,
 item_id BIGINT,
 behavior CHAR(2),
 `time` BIGINT
) WITH (
 'connector' = 'kafka',
 'topic' = 'user_behavior',
 'properties.bootstrap.servers' = 'localhost:9092',
 'properties.group.id' = 'user_behavior_flink',
 'format' = 'json',
 'json.ignore-parse-errors' = 'true',  
 'scan.startup.mode' = 'earliest-offset',
 'scan.topic-partition-discovery.interval' = '10000'
);

select * from UserBehavior;

 

i found same problem at http://apache-flink.147419.n8.nabble.com/sql-cli-sql-td7530.html

 

i see there are two dependencies conflicts

 

i try to solve the conflict, but it doesn't work

 

➜  flink-1.12-SNAPSHOT jar -tf ./lib/flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar |grep 'ConsumerRecord'
org/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecords$ConcatenatedIterable$1.class
org/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecord.class
org/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecords$ConcatenatedIterable.class
org/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecords.class
➜  flink-1.12-SNAPSHOT jar -tf ./lib/flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar |grep 'FlinkKafkaConsumer'
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.class
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase$1.class
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.class
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase$2.class
➜  flink-1.12-SNAPSHOT jar -tf ./lib/flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar |grep 'ByteArrayDeserializer'
org/apache/flink/kafka/shaded/org/apache/kafka/common/serialization/ByteArrayDeserializer.class
➜  flink-1.12-SNAPSHOT jar -tf ./lib/flink-sql-connector-kafka_2.11-1.12-SNAPSHOT.jar |grep 'FlinkKafkaConsumer'