Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1763

Handle broken symlinks when copying jars

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.9.0, 1.0.0
    • Fix Version/s: 1.0.0
    • Component/s: deployment
    • Labels:
      None

      Description

      When Hive is installed on a cluster and HBase isn't, init-hcfs.groovy will fail copying jars to HDFS for Oozie during puppet apply. This is because of a broken simlink in /usr/lib/hive/lib, namely hbase.jar which links to /usr/lib/hbase/hbase.jar.

      This is introduced by BIGTOP-1235.

      The exception thrown in init-hcfs.groovy:

      java.io.FileNotFoundException: File /usr/lib/hive/lib/hbase.jar does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:516)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:729)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:506)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:407)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1951)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1919)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1884)
        at org.apache.hadoop.fs.FileSystem$copyFromLocalFile$1.call(Unknown Source)
        at init-hcfs$_run_closure7_closure8.doCall(init-hcfs.groovy:258)
        at init-hcfs$_run_closure7.doCall(init-hcfs.groovy:252)
        at init-hcfs.run(init-hcfs.groovy:281)
      
      

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          Pushed. Thanks for the fix Peter!

          Show
          cos Konstantin Boudnik added a comment - Pushed. Thanks for the fix Peter!
          Hide
          cos Konstantin Boudnik added a comment -

          +1 - thanks for the fix. I will commit it shortly.

          Show
          cos Konstantin Boudnik added a comment - +1 - thanks for the fix. I will commit it shortly.
          Hide
          petersla Peter Slawski added a comment -

          Attached patch. I've verified that without HBase installed and Hive installed, init-hdfs.sh executes successfully with puppet apply:

          Debug: Exec[init hdfs](provider=posix): Executing 'bash -x /usr/lib/hadoop/libexec/init-hdfs.sh'
          Notice: /Stage[main]/Hadoop_head_node/Exec[init hdfs]/returns: executed successfully
          
          Show
          petersla Peter Slawski added a comment - Attached patch. I've verified that without HBase installed and Hive installed, init-hdfs.sh executes successfully with puppet apply: Debug: Exec[init hdfs](provider=posix): Executing 'bash -x /usr/lib/hadoop/libexec/init-hdfs.sh' Notice: /Stage[main]/Hadoop_head_node/Exec[init hdfs]/returns: executed successfully

            People

            • Assignee:
              petersla Peter Slawski
              Reporter:
              petersla Peter Slawski
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development