Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3317

Oozie sharedlib create command fails when uploading to HDFS

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0
    • Fix Version/s: 1.5.0, 1.4.1
    • Component/s: None
    • Labels:
      None

      Description

      Hi!

      I am testing oozie on BigTop 1.4, and when I execute the sharedlib create command I get the following error:

      /usr/bin/oozie-setup sharelib create -fs hdfs://analytics-test-hadoop -locallib /usr/lib/oozie/oozie-sharelib.tar.gz
      
      [..]
      
      the destination path for sharelib is: /user/oozie/share/lib/lib_20200228153329
      Error: No FileSystem for scheme: hdfs
      Stack trace for the error was (for debug purposes):--------------------------------------java.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:172) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:67)
      

      After some debugging I found the following broken symlink:

      file /usr/lib/oozie/lib/hadoop-hdfs.jar
      /usr/lib/oozie/lib/hadoop-hdfs.jar: broken symbolic link to ../../hadoop/client/hadoop-hdfs.jar
      

      The above is created by the oozie package, but the hadoop-hdfs.jar seems not present. I removed it and added the following in /usr/lib/oozie/lib:

      ln -s ../../hadoop/client/hadoop-hdfs-client.jar hadoop-hdfs-client.jar
      

      And now sharedlib upgrade seems working. It should be an easy fix in the deb packaging of oozie if what I wrote is correct (I can send a pull request in case).

        Attachments

          Activity

            People

            • Assignee:
              elukey Luca Toscano
              Reporter:
              elukey Luca Toscano

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 20m
                1h 20m

                  Issue deployment