Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1764

Fix copying mapreduce jars to HDFS for Oozie

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.0
    • Component/s: deployment
    • Labels:
      None

      Description

      On HDFS after puppet apply, /user/oozie/share/lib/distcp does not exist and /user/oozie/share/lib/mapreduce-streaming is empty. This looks to be a bug in copying jars in init-hcfs.groovy#L283

      $ hadoop fs -ls /user/oozie/share/lib/
      drwxr-xr-x   - hdfs hadoop          0 2015-03-18 21:47 /user/oozie/share/lib/distcp
      drwxr-xr-x   - hdfs hadoop          0 2015-03-18 21:47 /user/oozie/share/lib/hive
      drwxr-xr-x   - hdfs hadoop          0 2015-03-18 21:47 /user/oozie/share/lib/mapreduce
      drwxr-xr-x   - hdfs hadoop          0 2015-03-18 21:47 /user/oozie/share/lib/mapreduce-streaming
      drwxr-xr-x   - hdfs hadoop          0 2015-03-18 21:47 /user/oozie/share/lib/pig
      
      $ hadoop fs -ls /user/oozie/share/lib/mapreduce-streaming
      

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        I have noticed it in my testing, but attributed it to the fact that I didn't have Oozie installed on the cluster. Is yours installed?

        Show
        cos Konstantin Boudnik added a comment - I have noticed it in my testing, but attributed it to the fact that I didn't have Oozie installed on the cluster. Is yours installed?
        Hide
        petersla Peter Slawski added a comment -

        Yes, Oozie is installed on the cluster. It looks like the logic to copy jars isn't aware whether Oozie is installed or not.

        Show
        petersla Peter Slawski added a comment - Yes, Oozie is installed on the cluster. It looks like the logic to copy jars isn't aware whether Oozie is installed or not.
        Hide
        petersla Peter Slawski added a comment -

        Attached patch fixes the typos that cause this in init-hcfs.groovy. Thanks!

        Testing Done

        Verified that mapreduce jars are copied over after running puppet apply on a cluster:

        $ hadoop fs -ls /user/oozie/share/lib/distcp
        Found 2 items
        -rw-r--r--   3 hdfs hadoop      94055 2015-03-24 00:23 /user/oozie/share/lib/distcp/hadoop-distcp-2.6.0.jar
        -rw-r--r--   3 hdfs hadoop      94055 2015-03-24 00:23 /user/oozie/share/lib/distcp/hadoop-distcp.jar
        
        $ hadoop fs -ls /user/oozie/share/lib/mapreduce-streaming
        Found 2 items
        -rw-r--r--   3 hdfs hadoop     104990 2015-03-24 00:23 /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.6.0.jar
        -rw-r--r--   3 hdfs hadoop     104990 2015-03-24 00:23 /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming.jar
        
        Show
        petersla Peter Slawski added a comment - Attached patch fixes the typos that cause this in init-hcfs.groovy . Thanks! Testing Done Verified that mapreduce jars are copied over after running puppet apply on a cluster: $ hadoop fs -ls /user/oozie/share/lib/distcp Found 2 items -rw-r--r-- 3 hdfs hadoop 94055 2015-03-24 00:23 /user/oozie/share/lib/distcp/hadoop-distcp-2.6.0.jar -rw-r--r-- 3 hdfs hadoop 94055 2015-03-24 00:23 /user/oozie/share/lib/distcp/hadoop-distcp.jar $ hadoop fs -ls /user/oozie/share/lib/mapreduce-streaming Found 2 items -rw-r--r-- 3 hdfs hadoop 104990 2015-03-24 00:23 /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.6.0.jar -rw-r--r-- 3 hdfs hadoop 104990 2015-03-24 00:23 /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming.jar
        Hide
        cos Konstantin Boudnik added a comment -

        Great catch! +1 and I will commit it shortly.

        Show
        cos Konstantin Boudnik added a comment - Great catch! +1 and I will commit it shortly.
        Hide
        cos Konstantin Boudnik added a comment -

        Committed and pushed. Thanks Peter!

        Show
        cos Konstantin Boudnik added a comment - Committed and pushed. Thanks Peter!

          People

          • Assignee:
            petersla Peter Slawski
            Reporter:
            petersla Peter Slawski
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development