[FLINK-23449] YarnTaskExecutorRunner does not contains MapReduce classes

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.11.3
• Component/s: Connectors / Hive, Deployment / YARN
• Labels: pull-request-available

Environment

flink-1.11

flink on yarn cluster

jdk1.8

hive1.2.1

hadoop2.7

hadoop classes is provided with  export HADOOP_CLASSPATH=`hadoop classpath` when submitting test APP. （described in https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html ）

{{}}

• Assignee: Unassigned
• Reporter: Kai Chen
• Created: 21/Jul/21 10:34
• Updated: 08/Oct/21 11:29
• Resolved: 24/Aug/21 08:35

Description

I followed instructions described in https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive and tested hive streaming sink, met this exception  

      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf

http://apache-flink.147419.n8.nabble.com/Flink-td7866.html met the same problem.

 

I checked TM jvm envs and the code and found that flink only set up YARN_APPLICATION_CLASSPATH, but without MAPREDUCE_APPLICATION_CLASSPATH.

See: https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119

 

I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as spark does.

Issue Links

• is related to: FLINK-21319 hadoop-mapreduce jars are not loaded into classpath when submiting flink on yarn jobs.
• links to: GitHub Pull Request #16552