Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25253

An ClassNotFoundException of missing Hadoop class occurred when submitting to yarn

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 1.14.0
    • None
    • None
    • Environment version:
      Hadoop 3.1.1
      Hive 3.1.1
      Flink 1.4.0
      kafka 2.6.1

       

    Description

      I use the Hive Table connector to write hive from Kafka and submit to yarn successfully, but it will always report during execution
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
      Before submitting, I executed "export HADOOP_CLASSPATH=`hadoop classpath`" to import hadoop dependencies.

      I found in the JM log that the classpath already contains Hadoop dependencies, but an exception still occurs

      The original jar that I submitted only contains code and does not contain dependencies,The program loads dependencies from the hadoop path and lib under the flink directory

      The attachment contains flink lib, code, jar and JM log

      Thanks

      Attachments

        1. submit_log.log
          109 kB
          ghost synth
        2. pom.xml
          8 kB
          ghost synth
        3. original-TropicalaLink-1.0-SNAPSHOT.jar
          6 kB
          ghost synth
        4. FlinkPlaySubmit.scala
          3 kB
          ghost synth
        5. flink lib.png
          409 kB
          ghost synth

        Activity

          People

            Unassigned Unassigned
            ghostSynth ghost synth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 96h
                96h
                Remaining:
                Remaining Estimate - 96h
                96h
                Logged:
                Time Spent - Not Specified
                Not Specified