Apache Whirr (retired)
  1. Apache Whirr (retired)
  2. WHIRR-368

Add the ability to adjust contents of hadoop-env.sh from a cluster properties file

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.7.0
    • Component/s: core
    • Labels:

      Description

      I'd like to be able to adjust the values in the hadoop-env.sh via the .properties in a way similar to how adjusting settings in the hadoop configuration files is currently done.

      1. WHIRR-368-hbase-env.patch
        17 kB
        Karel Vervaeke
      2. WHIRR-368.patch
        32 kB
        Karel Vervaeke

        Issue Links

          Activity

          Hide
          Karel Vervaeke added a comment - - edited

          I have done this for the hbase master and regionserver java options in this branch: https://github.com/karel1980/whirr/tree/hbase-env

          Perhaps we should support arbitrary environment properties like this:

          whirr.hbase-env.properties.FOO=bar
          whirr.hadoop-env.properties.QUX=meta

          I don't think we should pass each of these as options to the configure_hbase and configure_hadoop scripts.
          Option one would be to generate the env.sh files locally and make them available via the blobcache.
          Option two would be to implement a function, and set the properties via individual calls:

          function set_env_var() {
            file="$1"
            name="$2"
            value="$3"
          
            # note: name and value should be escaped to avoid breakage
            sed -i -e "s|^export $name=|export $name=$value|" $file
          }
          

          I personally like the second one a little better, as it could be reused for different services.

          Show
          Karel Vervaeke added a comment - - edited I have done this for the hbase master and regionserver java options in this branch: https://github.com/karel1980/whirr/tree/hbase-env Perhaps we should support arbitrary environment properties like this: whirr.hbase-env.properties.FOO=bar whirr.hadoop-env.properties.QUX=meta I don't think we should pass each of these as options to the configure_hbase and configure_hadoop scripts. Option one would be to generate the env.sh files locally and make them available via the blobcache. Option two would be to implement a function, and set the properties via individual calls: function set_env_var() { file="$1" name="$2" value="$3" # note: name and value should be escaped to avoid breakage sed -i -e "s|^export $name=|export $name=$value|" $file } I personally like the second one a little better, as it could be reused for different services.
          Hide
          Tom White added a comment -

          WHIRR-370 would be another way of implementing this, and would be fairly generic across services.

          Show
          Tom White added a comment - WHIRR-370 would be another way of implementing this, and would be fairly generic across services.
          Hide
          Karel Vervaeke added a comment -

          Attached patch lets you append entries to hbase-env.sh using whirr.hbase-env.FOO=bar in the cluster properties file.

          Show
          Karel Vervaeke added a comment - Attached patch lets you append entries to hbase-env.sh using whirr.hbase-env.FOO=bar in the cluster properties file.
          Hide
          Karel Vervaeke added a comment -
            • The prefix used in the patch is 'hbase-env', not 'whirr.hbase-env'
          Show
          Karel Vervaeke added a comment - The prefix used in the patch is 'hbase-env', not 'whirr.hbase-env'
          Hide
          Tom White added a comment -

          I wonder if it's simpler to generate the whole hbase-env.sh file on the client and copy it to /tmp on the cloud machine, just like we do for hbase-site.xml?

          Also, can you put the default values of HBASE_MASTER_OPTS and HBASE_REGIONSERVER_OPTS in whirr-hbase-default.properties, rather than in the Java class? This would avoid having special cases in HBaseClusterActionHandler#addHBaseEnvParams().

          Show
          Tom White added a comment - I wonder if it's simpler to generate the whole hbase-env.sh file on the client and copy it to /tmp on the cloud machine, just like we do for hbase-site.xml? Also, can you put the default values of HBASE_MASTER_OPTS and HBASE_REGIONSERVER_OPTS in whirr-hbase-default.properties, rather than in the Java class? This would avoid having special cases in HBaseClusterActionHandler#addHBaseEnvParams().
          Hide
          Karel Vervaeke added a comment -

          I'm ok with generating it on the client first, but should we copy it over the existing file or append?
          When we copy over it, we should also include HBASE_OPTS (which has changed between hbase 0.89.x and 0.90.x in the default hbase-env.sh)

          Show
          Karel Vervaeke added a comment - I'm ok with generating it on the client first, but should we copy it over the existing file or append? When we copy over it, we should also include HBASE_OPTS (which has changed between hbase 0.89.x and 0.90.x in the default hbase-env.sh)
          Hide
          Karel Vervaeke added a comment -

          Updated patch: hadoop-env and hbase-env are generated based on the cluster properties file, with defaults in whirr-

          {hadoop,hbase}

          -default.properties.

          Show
          Karel Vervaeke added a comment - Updated patch: hadoop-env and hbase-env are generated based on the cluster properties file, with defaults in whirr- {hadoop,hbase} -default.properties.
          Hide
          Karel Vervaeke added a comment -

          Ready for final review

          Show
          Karel Vervaeke added a comment - Ready for final review
          Hide
          Andrei Savu added a comment -

          +1 code looks good and integration tests are passing on cloudservers-us

          Show
          Andrei Savu added a comment - +1 code looks good and integration tests are passing on cloudservers-us
          Hide
          Tom White added a comment -

          +1 looks good. Thanks for updating it Karel.

          Show
          Tom White added a comment - +1 looks good. Thanks for updating it Karel.
          Hide
          Andrei Savu added a comment -

          I have just committed this. Thanks Karel for making this happen. Thanks Tom for reviewing.

          Show
          Andrei Savu added a comment - I have just committed this. Thanks Karel for making this happen. Thanks Tom for reviewing.
          Hide
          Karel Vervaeke added a comment -

          Heh. I just did svn up; patch; svn commit and got merge conflicts from you applying the same patch

          Show
          Karel Vervaeke added a comment - Heh. I just did svn up; patch; svn commit and got merge conflicts from you applying the same patch
          Hide
          Andrei Savu added a comment -

          I won the race this time.

          Show
          Andrei Savu added a comment - I won the race this time.

            People

            • Assignee:
              Karel Vervaeke
              Reporter:
              Soren Macbeth
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development