[FLINK-32418] ClassNotFoundException when using flink-protobuf with sql-client

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.2
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Client
• Labels: pull-request-available stale-assigned
• Assignee: Michael Kreis
• Reporter: Michael Kreis
• Created: 23/Jun/23 05:42
• Updated: 21/Feb/24 02:22

Description

When the protobuf format in the kafka connector is used via the sql-client it is not able to load the generated protobuf classes which are either passed via `-j /protobuf-classes.jar` or added in the script via ADD JAR '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded but when the protobuf classes are loaded a ClassNotFoundException occurs.

executed command:

sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar

protobuf-table.sql

ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';

SHOW JARS;

CREATE TABLE POSITIONS(id BIGINT) WITH (
      'connector' = 'kafka',
      'format' = 'protobuf',
      'topic' = 'protbuf-topic',
      'properties.bootstrap.servers' = 'kafka:9092',
      'properties.group.id' = 'flink-protobuf',
      'properties.security.protocol' = 'SASL_PLAINTEXT',
      'properties.sasl.mechanism' = 'SCRAM-SHA-512',
      'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="****";',
      'scan.startup.mode' = 'earliest-offset',
      'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
      'protobuf.ignore-parse-errors' = 'true'
      );

SELECT * FROM POSITIONS;

exception in the log:

Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Unknown Source)
        at org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
        ... 36 more

This also seems somehow related to FLINK-30318

Issue Links

• is duplicated by: FLINK-34472 loading class of protobuf format descriptor by "Class.forName(className, true, Thread.currentThread().getContextClassLoader())" may can not find class because the current thread class loader may not contain this class
• is related to: FLINK-30318 sql-client failed to load jar passed with -j args in flink 1.16.0
• links to: GitHub Pull Request #23032