Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2787

Oozie distributes application jar twice making the spark job fail

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.1
    • Component/s: None
    • Labels:
      None

      Description

      Oozie adds the application jar to the list of files to be uploaded to distributed cache. Since this gets added twice, the job fails. This is observed from spark 2.1.0 which introduces a check for same file and fails the job.

      --master
      yarn
      --deploy-mode
      cluster
      --name
      oozieSparkStarter
      --class
      ScalaWordCount
      --queue 
      default
      --conf
      spark.executor.extraClassPath=$PWD/*
      --conf
      spark.driver.extraClassPath=$PWD/*
      --conf
      spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
      --conf
      spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
      --conf
      spark.yarn.security.tokens.hive.enabled=false
      --conf
      spark.yarn.security.tokens.hbase.enabled=false
      --files
      hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
      --properties-file
      spark-defaults.conf
      --verbose
      spark-example.jar
      samplefile.txt
      output
      

        Attachments

        1. OOZIE-2787-1.patch
          3 kB
          Satish Subhashrao Saley
        2. OOZIE-2787-2.patch
          9 kB
          Satish Subhashrao Saley
        3. OOZIE-2787-3.patch
          10 kB
          Satish Subhashrao Saley
        4. OOZIE-2787-4.patch
          10 kB
          Satish Subhashrao Saley
        5. OOZIE-2787-5.patch
          12 kB
          Satish Subhashrao Saley
        6. OOZIE-2787-amend-1.patch
          2 kB
          Satish Subhashrao Saley
        7. OOZIE-2787-amend-2.patch
          10 kB
          Satish Subhashrao Saley
        8. OOZIE-2787-amend-3.patch
          10 kB
          Satish Subhashrao Saley
        9. OOZIE-2787-amend-4.patch
          10 kB
          Satish Subhashrao Saley
        10. OOZIE-2787-amend-5.patch
          10 kB
          Satish Subhashrao Saley

          Issue Links

            Activity

              People

              • Assignee:
                satishsaley Satish Subhashrao Saley
                Reporter:
                satishsaley Satish Subhashrao Saley
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: