Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2802

Spark action failure on Spark 2.1.0 due to duplicate sharelibs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0.0b1, 4.3.1
    • Component/s: None
    • Labels:
      None

      Description

      There are some same jars in launcher lib and sharelib. These jars are causing failures in spark action for spark 2.1.0. SPARK-18099 added an exception to be thrown if same file gets added multiple times to distributed cache.

      $ hdfs dfs -ls /user/saley/share/lib/launcher_20170216150752/oozie
      17/02/16 15:14:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Found 3 items
      -rwxr-xr-x   1 saley supergroup      11871 2017-02-16 15:07 /user/saley/share/lib/launcher_20170216150752/oozie/oozie-hadoop-utils-hadoop-2-4.4.0-SNAPSHOT.jar
      -rwxr-xr-x   1 saley supergroup      17514 2017-02-16 15:07 /user/saley/share/lib/launcher_20170216150752/oozie/oozie-sharelib-hcatalog-4.4.0-SNAPSHOT.jar
      -rwxr-xr-x   1 saley supergroup      55092 2017-02-16 15:07 /user/saley/share/lib/launcher_20170216150752/oozie/oozie-sharelib-oozie-4.4.0-SNAPSHOT.jar
      
      
      
      $ hdfs dfs -ls /user/saley/share/lib/lib_20170216150706/oozie
      17/02/16 15:14:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Found 3 items
      -rw-r--r--   1 saley supergroup      16046 2017-02-16 15:07 /user/saley/share/lib/lib_20170216150706/oozie/json-simple-1.1.jar
      -rw-r--r--   1 saley supergroup      11871 2017-02-16 15:07 /user/saley/share/lib/lib_20170216150706/oozie/oozie-hadoop-utils-hadoop-2-4.4.0-SNAPSHOT.jar
      -rw-r--r--   1 saley supergroup      55092 2017-02-16 15:07 /user/saley/share/lib/lib_20170216150706/oozie/oozie-sharelib-oozie-4.4.0-SNAPSHOT.jar
      
      

      The file with same name can't be in twice in any of --files , --archives, --jars (ie one file in both --files and --jars) and can't be in --files twice and can't be in --archives twice. It is allowed to be in --jars twice.

      Possible solutions:
      1. While specifying the --files options, filter out the duplicate jars
      2. Separate out jars and files. Use --jars options to distribute jars and --files for files. Also remember to remove duplicates from --files, and make sure that there is nothing common in --jars and --files.

        Attachments

        1. OOZIE-2802-001.patch
          4 kB
          Peter Cseh
        2. OOZIE-2802-002.patch
          10 kB
          Peter Cseh
        3. OOZIE-2802-003.patch
          10 kB
          Peter Cseh
        4. spark-2.1-failure-launcherlib.txt
          158 kB
          Satish Subhashrao Saley

          Issue Links

            Activity

              People

              • Assignee:
                gezapeti Peter Cseh
                Reporter:
                satishsaley Satish Subhashrao Saley
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: