Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.6.0
    • Component/s: general
    • Labels:
      None

      Description

      It would be nice to have a single script that can either setup or check the key HDFS locations and their permissions (things like /tmp, /user, /user/joe, /mapred, /hbase, etc.). We already have some of that functionality in our puppet, but it would be useful to provide it to the users of the cluster.

      1. BIGTOP-547.1.patch
        5 kB
        Mark Grover
      2. BIGTOP-547.2.patch
        6 kB
        Mark Grover

        Issue Links

          Activity

          Hide
          Mark Grover added a comment -

          +1 on the idea. I am stealing this from you, Roman!

          Show
          Mark Grover added a comment - +1 on the idea. I am stealing this from you, Roman!
          Hide
          Roman Shaposhnik added a comment -

          Great! Would love to see it implemented. Also, it would be very nice if we can incorporate the local -> HDFS jar rsyncing that is happening for Oozie's benefit. Here's the list of stuff Oozie cares about: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifests/cluster.pp#L242

          Show
          Roman Shaposhnik added a comment - Great! Would love to see it implemented. Also, it would be very nice if we can incorporate the local -> HDFS jar rsyncing that is happening for Oozie's benefit. Here's the list of stuff Oozie cares about: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifests/cluster.pp#L242
          Hide
          Mark Grover added a comment -

          Sure, will do. Thanks!

          Show
          Mark Grover added a comment - Sure, will do. Thanks!
          Hide
          Mark Grover added a comment -

          First version of the patch.

          I would appreciate feedback especially along the lines of:
          1. The location and name of the script. I created a new directory called bigtop-deploy/manual and named the file init-hdfs.sh. "manual" is most likely not the best term to describe what a by-hand install so I am open to suggestions.

          2. I kept the script pretty dumb. No fancy associative arrays that are looped through to create the appropriate HDFS directories and the permissions like the puppet code does. If someone feels strongly about that, let me know, I can make it more flashy

          Show
          Mark Grover added a comment - First version of the patch. I would appreciate feedback especially along the lines of: 1. The location and name of the script. I created a new directory called bigtop-deploy/manual and named the file init-hdfs.sh. "manual" is most likely not the best term to describe what a by-hand install so I am open to suggestions. 2. I kept the script pretty dumb. No fancy associative arrays that are looped through to create the appropriate HDFS directories and the permissions like the puppet code does. If someone feels strongly about that, let me know, I can make it more flashy
          Hide
          Peter Linnell added a comment -

          1. How about bigtop-deploy/setup ?

          2. Simple != dumb. It looks readable and maintainable to me.

          This is a great idea. It automates one of the more time consuming/error prone steps in setting up hadoop.

          My only question is /bin/sh vs bin/bash. I thought we've kind of standardized on bash, but I also know *buntu flavors have dash as the default.

          Show
          Peter Linnell added a comment - 1. How about bigtop-deploy/setup ? 2. Simple != dumb. It looks readable and maintainable to me. This is a great idea. It automates one of the more time consuming/error prone steps in setting up hadoop. My only question is /bin/sh vs bin/bash. I thought we've kind of standardized on bash, but I also know *buntu flavors have dash as the default.
          Hide
          Roman Shaposhnik added a comment -

          Mark, it looks good as a first cut. One question though – wouldn't it be more useful to package this script up and make it available to all HDFS users?

          Show
          Roman Shaposhnik added a comment - Mark, it looks good as a first cut. One question though – wouldn't it be more useful to package this script up and make it available to all HDFS users?
          Hide
          Mark Grover added a comment -

          Thanks Peter Linnell for the review. The patch does specify "/bin/bash", where do you see a reference to "/bin/sh"?

          Thanks Roman Shaposhnik. I do like the idea of packaging it. Is there an existing package that would be a good place for this script? Or, are you anticipating more helper scripts like these and consequently thinking about creating a new package?

          Show
          Mark Grover added a comment - Thanks Peter Linnell for the review. The patch does specify "/bin/bash", where do you see a reference to "/bin/sh"? Thanks Roman Shaposhnik . I do like the idea of packaging it. Is there an existing package that would be a good place for this script? Or, are you anticipating more helper scripts like these and consequently thinking about creating a new package?
          Hide
          Roman Shaposhnik added a comment -

          I'd say it can go into hadoop-hdfs directly. After all, we already have /usr/bin/hdfs &co helper scripts there.

          Show
          Roman Shaposhnik added a comment - I'd say it can go into hadoop-hdfs directly. After all, we already have /usr/bin/hdfs &co helper scripts there.
          Hide
          Mark Grover added a comment -

          That sounds good to me. Let me create a new patch with addition of this script to hadoop-hdfs package.

          Show
          Mark Grover added a comment - That sounds good to me. Let me create a new patch with addition of this script to hadoop-hdfs package.
          Hide
          Sean Mackrory added a comment -

          I second the support of simplicity. There's a lot of systems out there that still don't have a version of bash that supports associative arrays. Simplicity tends to help portability.

          Show
          Sean Mackrory added a comment - I second the support of simplicity. There's a lot of systems out there that still don't have a version of bash that supports associative arrays. Simplicity tends to help portability.
          Hide
          Mark Grover added a comment -

          New patch where the script is included in hadoop-hdfs package.

          Show
          Mark Grover added a comment - New patch where the script is included in hadoop-hdfs package.
          Hide
          Peter Linnell added a comment -

          +1 Thanks for this.

          Show
          Peter Linnell added a comment - +1 Thanks for this.
          Hide
          Johnny Zhang added a comment -

          +1, make sense to me

          Show
          Johnny Zhang added a comment - +1, make sense to me
          Hide
          Roman Shaposhnik added a comment -

          +1 and committed!

          Show
          Roman Shaposhnik added a comment - +1 and committed!
          Hide
          Mark Grover added a comment -

          Thanks Roman for committing this! I created a follow-up JIRA BIGTOP-852.

          Show
          Mark Grover added a comment - Thanks Roman for committing this! I created a follow-up JIRA BIGTOP-852 .

            People

            • Assignee:
              Mark Grover
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development