[FLINK-35064] Flink sql connector pulsar/hive com.fasterxml.jackson.annotation.JsonFormat$Value conflict

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.1
• Component/s: Connectors / Hive, Connectors / Pulsar
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: elon_X
• Created: 09/Apr/24 08:55
• Updated: 15/Apr/24 07:57

Description

When I compile and package flink-sql-connector-pulsar & flink-sql-connector-hive, and then put these two jar files into the Flink lib directory, I execute the following SQL statement through bin/sql-client.sh:

 

// code placeholder
CREATE TABLE
pulsar_table (
content string,
proc_time AS PROCTIME ()
)
WITH
(
'connector' = 'pulsar',
'topics' = 'persistent://xxx',
'service-url' = 'pulsar://xxx',
'source.subscription-name' = 'xxx',
'source.start.message-id' = 'latest',
'format' = 'csv',
'pulsar.client.authPluginClassName' = 'org.apache.pulsar.client.impl.auth.AuthenticationToken',
'pulsar.client.authParams' = 'token:xxx'
);
 
select * from pulsar_table; 

The task error exception stack is as follows：

 

Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.annotation.JsonFormat$Value.empty()Lcom/fasterxml/jackson/annotation/JsonFormat$Value;    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.cfg.MapperConfig.<clinit>(MapperConfig.java:56) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:660) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:576) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.common.util.ObjectMapperFactory.createObjectMapperInstance(ObjectMapperFactory.java:151) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.common.util.ObjectMapperFactory.<clinit>(ObjectMapperFactory.java:142) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.create(ConfigurationDataUtils.java:35) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.<clinit>(ConfigurationDataUtils.java:43) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.pulsar.client.impl.ClientBuilderImpl.loadConf(ClientBuilderImpl.java:77) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.flink.connector.pulsar.common.config.PulsarClientFactory.createClient(PulsarClientFactory.java:105) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.<init>(PulsarSourceEnumerator.java:95) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.<init>(PulsarSourceEnumerator.java:76) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.flink.connector.pulsar.source.PulsarSource.createEnumerator(PulsarSource.java:144) ~[flink-sql-connector-pulsar-4.0-SNAPSHOT.jar:4.0-SNAPSHOT]    at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:213) ~[flink-dist_2.12-1.16.1.jar:1.16.1]

 

The exception shows a conflict with com.fasterxml.jackson.annotation.JsonFormat$Value. I investigated and found that flink-sql-connector-pulsar and flink-sql-connector-hive depend on different versions, leading to this conflict.

// flink-sql-connector-pulsar pom.xml
<dependency>
    <groupId>com.fasterxml.jackson</groupId>
    <artifactId>jackson-bom</artifactId>
    <type>pom</type>
    <scope>import</scope>
    <version>2.13.4.20221013</version>
</dependency> 

// flink-sql-connector-hive pom.xml
<dependency>
    <groupId>com.fasterxml.jackson</groupId>
    <artifactId>jackson-bom</artifactId>
    <type>pom</type>
    <scope>import</scope>
    <version>2.15.3</version>
</dependency>

 

Issue Links

• links to: GitHub Pull Request #88