[FLINK-25253] An ClassNotFoundException of missing Hadoop class occurred when submitting to yarn

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: Connectors / Hive, Deployment / YARN, Documentation

Environment

Environment version:
Hadoop 3.1.1
Hive 3.1.1
Flink 1.4.0
kafka 2.6.1

 

Language

scala

• Assignee: Unassigned
• Reporter: ghost synth
• Created: 10/Dec/21 09:15
• Updated: 20/Dec/21 12:25

Description

I use the Hive Table connector to write hive from Kafka and submit to yarn successfully, but it will always report during execution
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
Before submitting, I executed "export HADOOP_CLASSPATH=`hadoop classpath`" to import hadoop dependencies.

I found in the JM log that the classpath already contains Hadoop dependencies, but an exception still occurs

The original jar that I submitted only contains code and does not contain dependencies,The program loads dependencies from the hadoop path and lib under the flink directory

The attachment contains flink lib, code, jar and JM log

Thanks