HBase
  1. HBase
  2. HBASE-4415

Add configuration script for setup HBase (hbase-setup-conf.sh)

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: 0.90.4, 0.92.0
    • Fix Version/s: 0.92.3
    • Component/s: scripts
    • Labels:
      None
    • Environment:

      Java 6, Linux

      Description

      The goal of this jura is to provide a installation script for configuring HBase environment and configuration. By using the same pattern of *-setup-conf.sh for all Hadoop related projects. For HBase, the usage of the script looks like this:

      usage: ./hbase-setup-conf.sh <parameters>
      
        Optional parameters:
          --hadoop-conf=/etc/hadoop            Set Hadoop configuration directory location
          --hadoop-home=/usr                       Set Hadoop directory location
          --hadoop-namenode=localhost              Set Hadoop namenode hostname
          --hadoop-replication=3                   Set HDFS replication
          --hbase-home=/usr                        Set HBase directory location
          --hbase-conf=/etc/hbase                  Set HBase configuration directory location
          --hbase-log=/var/log/hbase               Set HBase log directory location
          --hbase-pid=/var/run/hbase               Set HBase pid directory location
          --hbase-user=hbase                       Set HBase user
          --java-home=/usr/java/default            Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM    Set Kerberos realm
          --kerberos-principal-id=_HOST            Set Kerberos principal ID 
          --keytab-dir=/etc/security/keytabs       Set keytab directory
          --regionservers=localhost                Set regionservers hostnames
          --zookeeper-home=/usr                    Set ZooKeeper directory location
          --zookeeper-quorum=localhost             Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
      
      1. HBASE-4415-9.patch
        20 kB
        Arpit Gupta
      2. HBASE-4415-8.patch
        18 kB
        Giridharan Kesavan
      3. HBASE-4415-7.patch
        3 kB
        Giridharan Kesavan
      4. HBASE-4415-6.patch
        18 kB
        Eric Yang
      5. HBASE-4415-5.patch
        18 kB
        Eric Yang
      6. HBASE-4415-4.patch
        18 kB
        Eric Yang
      7. HBASE-4415-3.patch
        17 kB
        Eric Yang
      8. HBASE-4415-2.patch
        17 kB
        Eric Yang
      9. HBASE-4415-1.patch
        16 kB
        Eric Yang
      10. HBASE-4415.patch
        16 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Eric Yang added a comment -

          HBase setup script with configuration templates

          Show
          Eric Yang added a comment - HBase setup script with configuration templates
          Hide
          Eric Yang added a comment -

          Fixed hadoop-conf location.

          Show
          Eric Yang added a comment - Fixed hadoop-conf location.
          Hide
          Eric Yang added a comment -

          Fixed usage screen hadoop-conf location.

          Show
          Eric Yang added a comment - Fixed usage screen hadoop-conf location.
          Hide
          Eric Yang added a comment -
          Show
          Eric Yang added a comment - Review Board URL: https://reviews.apache.org/r/1925/
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:
          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location
          --hadoop-home=/usr Set Hadoop directory location
          --hadoop-namenode=localhost Set Hadoop namenode hostname
          --hadoop-replication=3 Set HDFS replication
          --hbase-home=/usr Set HBase directory location
          --hbase-conf=/etc/hbase Set HBase configuration directory location
          --hbase-log=/var/log/hbase Set HBase log directory location
          --hbase-pid=/var/run/hbase Set HBase pid directory location
          --hbase-user=hbase Set HBase user
          --java-home=/usr/java/default Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
          --kerberos-principal-id=_HOST Set Kerberos principal ID
          --keytab-dir=/etc/security/keytabs Set keytab directory
          --regionservers=localhost Set regionservers hostnames
          --zookeeper-home=/usr Set ZooKeeper directory location
          --zookeeper-quorum=localhost Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.
          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs


          /src/assembly/all.xml 1170899
          /src/packages/hbase-setup-conf.sh PRE-CREATION
          /src/packages/templates/conf/hbase-env.sh PRE-CREATION
          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing
          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- Review request for hbase. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs /src/assembly/all.xml 1170899 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/
          -----------------------------------------------------------

          (Updated 2011-09-16 00:06:21.722559)

          Review request for hbase.

          Summary (updated)
          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:
          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location
          --hadoop-home=/usr Set Hadoop directory location
          --hadoop-namenode=localhost Set Hadoop namenode hostname
          --hadoop-replication=3 Set HDFS replication
          --hbase-home=/usr Set HBase directory location
          --hbase-conf=/etc/hbase Set HBase configuration directory location
          --hbase-log=/var/log/hbase Set HBase log directory location
          --hbase-pid=/var/run/hbase Set HBase pid directory location
          --hbase-user=hbase Set HBase user
          --java-home=/usr/java/default Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
          --kerberos-principal-id=_HOST Set Kerberos principal ID
          --keytab-dir=/etc/security/keytabs Set keytab directory
          --regionservers=localhost Set regionservers hostnames
          --zookeeper-home=/usr Set ZooKeeper directory location
          --zookeeper-quorum=localhost Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.
          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs


          /src/assembly/all.xml 1170899
          /src/packages/hbase-setup-conf.sh PRE-CREATION
          /src/packages/templates/conf/hbase-env.sh PRE-CREATION
          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing
          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-16 00:06:21.722559) Review request for hbase. Summary (updated) ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs /src/assembly/all.xml 1170899 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/#review1957
          -----------------------------------------------------------

          Generally it might be good to be able to define prefixes similar to what autoconf scripts do. Such as:
          --prefix=DIR [PREFIX/etc]
          --sysconfdir=DIR [PREFIX/etc]

          And so on.

          The hbase-site.xml and hbase-env.sh templates now have to be maintained separately, hence this makes config changes harder to maintain.

          In general this seems to be mostly useful for beginners of HBase, right?
          (Oldtimers will have a mechanism in place to setup HBase - puppet, chef, fabric, etc.

          So I wonder if a more useful avenue would be to post this as part of the HBase book or some other getting started spot on the HBase site.

          /src/packages/hbase-setup-conf.sh
          <https://reviews.apache.org/r/1925/#comment4436>

          Isn't /usr a strange default for a Hadoop (or HBase or ZooKeeper) installation?
          Maybe the script should rather check for a value, and fail if none was given.

          /src/packages/hbase-setup-conf.sh
          <https://reviews.apache.org/r/1925/#comment4437>

          I don't see HBASE_LOG_DIR defaulted anywhere.
          Same for HBASE_PID_DIR.

          /src/packages/templates/conf/hbase-env.sh
          <https://reviews.apache.org/r/1925/#comment4438>

          Wow 2007... that's old
          I think the latest convention is not to include he year.

          • Lars

          On 2011-09-16 00:06:21, Eric Yang wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1925/

          -----------------------------------------------------------

          (Updated 2011-09-16 00:06:21)

          Review request for hbase.

          Summary

          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:

          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location

          --hadoop-home=/usr Set Hadoop directory location

          --hadoop-namenode=localhost Set Hadoop namenode hostname

          --hadoop-replication=3 Set HDFS replication

          --hbase-home=/usr Set HBase directory location

          --hbase-conf=/etc/hbase Set HBase configuration directory location

          --hbase-log=/var/log/hbase Set HBase log directory location

          --hbase-pid=/var/run/hbase Set HBase pid directory location

          --hbase-user=hbase Set HBase user

          --java-home=/usr/java/default Set JAVA_HOME directory location

          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm

          --kerberos-principal-id=_HOST Set Kerberos principal ID

          --keytab-dir=/etc/security/keytabs Set keytab directory

          --regionservers=localhost Set regionservers hostnames

          --zookeeper-home=/usr Set ZooKeeper directory location

          --zookeeper-quorum=localhost Set ZooKeeper Quorum

          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.

          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs

          -----

          /src/assembly/all.xml 1170899

          /src/packages/hbase-setup-conf.sh PRE-CREATION

          /src/packages/templates/conf/hbase-env.sh PRE-CREATION

          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing

          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/#review1957 ----------------------------------------------------------- Generally it might be good to be able to define prefixes similar to what autoconf scripts do. Such as: --prefix=DIR [PREFIX/etc] --sysconfdir=DIR [PREFIX/etc] And so on. The hbase-site.xml and hbase-env.sh templates now have to be maintained separately, hence this makes config changes harder to maintain. In general this seems to be mostly useful for beginners of HBase, right? (Oldtimers will have a mechanism in place to setup HBase - puppet, chef, fabric, etc. So I wonder if a more useful avenue would be to post this as part of the HBase book or some other getting started spot on the HBase site. /src/packages/hbase-setup-conf.sh < https://reviews.apache.org/r/1925/#comment4436 > Isn't /usr a strange default for a Hadoop (or HBase or ZooKeeper) installation? Maybe the script should rather check for a value, and fail if none was given. /src/packages/hbase-setup-conf.sh < https://reviews.apache.org/r/1925/#comment4437 > I don't see HBASE_LOG_DIR defaulted anywhere. Same for HBASE_PID_DIR. /src/packages/templates/conf/hbase-env.sh < https://reviews.apache.org/r/1925/#comment4438 > Wow 2007... that's old I think the latest convention is not to include he year. Lars On 2011-09-16 00:06:21, Eric Yang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-16 00:06:21) Review request for hbase. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs ----- /src/assembly/all.xml 1170899 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          stack added a comment -

          Will the below break what we have now?

               <fileSet>
          +      <directory>src/packages/templates/conf</directory>
          +      <outputDirectory>share/hbase/templates/conf</outputDirectory>
          +    </fileSet>
          +    <fileSet>
          

          Do we have to reproduce hbase-env.sh over in src/packages? Similar for hbase-site.xml.

          Can you point me at why src/packages. Is this from hadoop and its src for building rpms etc.?

          So, how would i use this new script?

          Show
          stack added a comment - Will the below break what we have now? <fileSet> + <directory>src/packages/templates/conf</directory> + <outputDirectory>share/hbase/templates/conf</outputDirectory> + </fileSet> + <fileSet> Do we have to reproduce hbase-env.sh over in src/packages? Similar for hbase-site.xml. Can you point me at why src/packages. Is this from hadoop and its src for building rpms etc.? So, how would i use this new script?
          Hide
          Eric Yang added a comment -

          share/hbase/templates/conf is an newly added template directory. People can still use the default files in conf directory for backward compatibility until hbase-setup-conf.sh become the standard method of configuring HBase cluster. For now, there are two copies of hbase-env.sh, one in conf directory, and one in src/packages/templates/conf. The copy in src/packages/templates/conf is designed to be consumed by hbase-setup-conf.sh, and it mimics the changes that happens to trunk/conf/hbase-env.sh. If the community agrees that we can use one copy of base-env.sh and hbase-site.xml for both purpose of script automation and hand edit config file. I am more than willing to unify them as one copy and place in src/packages/templates/conf for clarity to indicate this is for script automation.

          Standard Maven practice uses the following structure:

          src/main - application code
          src/test - test code
          

          In Hadoop related projects, release packaging has introduced new directory structure:

          src/packages/tarball - tarball manifest file or related meta data
          src/packages/rpm - rpm related meta data
          src/packages/deb - debian related meta data
          src/packages/template/conf - Configuration template files
          

          This practice is to improve the release packages. In the past, Hadoop related projects are releasing a binary/source hybrid tar ball. This has proven problematic to maintain for different architecture. The intention is to improve the system so that we can create binary release for each architecture and decouple source and binary from tar ball releases. There are some more work to be done for packaging separation, but it is not in scope for this jira.

          The script is designed to run as root. For setting up a hbase cluster with hadoop 0.20.205. There are a few assumption made for kerberos key tabs. The key tab file names should be:

          namenode - nn.service.keytab
          secondarynamenode - sn.service.keytab
          datanode - dn.service.keytab
          jobtracker - jt.service.keytab
          tasktracker - tt.service.keytab
          hbase - hbase.service.keytab

          Compose a shell script that invokes each component's installation script and run them through pdsh, or use Apache Ambari (HMS):

          export namenode=nn1
          export datanode=dn1,dn2,dn3,dn4
          export secondary_namenode=sn1
          export jobtracker=jt1
          export tasktracker=tt1,tt2,tt3,tt4
          export hbase_regionservers=rs1,rs2,rs3,rs4
          export hbase_master=hbm
          
          # Setup Hadoop configuration
          /home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-conf.sh \
            --java-home=/home/hms/apps/jdk1.6.0_27 \
            --namenode-host=${namenode} \
            --secondarynamenode-host=${secondary_namenode} \
            --jobtracker-host=${jobtracker} \
            --conf-dir=/etc/hadoop \
            --hdfs-dir=/data/0/hadoop/var/hdfs,/data/1/hadoop/var/hdfs,/data/2/hadoop/var/hdfs,/data/3/hadoop/var/hdfs \
            --namenode-dir=/data/2/hadoop/var/hdfs/name \
            --mapred-dir=/data/0/tmp/mapred-local,/data/1/tmp/mapred-local,/data/2/tmp/mapred-local,/data/3/tmp/mapred-local \
            --datanode-dir=/data/0/hadoop/var/hdfs/data,/data/1/hadoop/var/hdfs/data,/data/2/hadoop/var/hdfs/data,/data/3/hadoop/var/hdfs/data \
            --log-dir=/home/hms/apps/hadoop-0.20.205.0/logs \
            --pid-dir=/home/hms/apps/hadoop-0.20.205.0/logs \
            --hdfs-user=hdfs \
            --mapreduce-user=mr \
            --kerberos-realm=APACHE.ORG \
            --hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \
            --mapreduce-user-keytab=/home/mr/mr.headless.keytab \
            --datanodes=${datanode} \
            --tasktrackers=${tasktracker} \
            --group=hadoop \
            --auto
          
          # Format Hadoop HDFS
          /home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-hdfs.sh \
            --config /etc/hadoop \
            --hdfs-user=hrt_hdfs \
            --mapreduce-user=hrt_mr \
            --kerberos-realm=APACHE.ORG \
            --hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \
            --format
          
          # Start up secondary name node, datanodes, job tracker, task tracker through hadoop-daemon.sh
          
          # Setup HBase
          /home/hms/apps/hbase-0.90.3/sbin/hbase-setup-conf.sh \
            --hadoop-conf=/etc/hadoop \
            --hadoop-home=/home/hms/apps/hadoop-0.20.205.0 \
            --hadoop-namenode=${namenode} \
            --hbase-home=/home/hms/apps/hbase-0.90.3 \
            --hbase-conf=/home/hms/apps/hbase-0.90.3/conf \
            --hbase-log=/home/hms/apps/hbase-0.90.3/logs \
            --hbase-pid=/home/hms/apps/hbase-0.90.3/var/run \
            --java-home=/usr/java/default \
            --kerberos-realm=APACHE.ORG \
            --kerberos-principal-id=hbase/_HOST \
            --keytab-dir=/etc/security/keytabs \
            --regionservers=${hbase_regionservers} \
            --zookeeper-quorum=${hbase_master} \
            --zookeeper-snapshot=/home/hms/apps/var/lib/zookeeper
            --hbase-user=hbase
            --hadoop-replication=3
          
          # Start HBase Master, region servers using hbase-daemon.sh.
          

          Hope this helps.

          Show
          Eric Yang added a comment - share/hbase/templates/conf is an newly added template directory. People can still use the default files in conf directory for backward compatibility until hbase-setup-conf.sh become the standard method of configuring HBase cluster. For now, there are two copies of hbase-env.sh, one in conf directory, and one in src/packages/templates/conf. The copy in src/packages/templates/conf is designed to be consumed by hbase-setup-conf.sh, and it mimics the changes that happens to trunk/conf/hbase-env.sh. If the community agrees that we can use one copy of base-env.sh and hbase-site.xml for both purpose of script automation and hand edit config file. I am more than willing to unify them as one copy and place in src/packages/templates/conf for clarity to indicate this is for script automation. Standard Maven practice uses the following structure: src/main - application code src/test - test code In Hadoop related projects, release packaging has introduced new directory structure: src/packages/tarball - tarball manifest file or related meta data src/packages/rpm - rpm related meta data src/packages/deb - debian related meta data src/packages/template/conf - Configuration template files This practice is to improve the release packages. In the past, Hadoop related projects are releasing a binary/source hybrid tar ball. This has proven problematic to maintain for different architecture. The intention is to improve the system so that we can create binary release for each architecture and decouple source and binary from tar ball releases. There are some more work to be done for packaging separation, but it is not in scope for this jira. The script is designed to run as root. For setting up a hbase cluster with hadoop 0.20.205. There are a few assumption made for kerberos key tabs. The key tab file names should be: namenode - nn.service.keytab secondarynamenode - sn.service.keytab datanode - dn.service.keytab jobtracker - jt.service.keytab tasktracker - tt.service.keytab hbase - hbase.service.keytab Compose a shell script that invokes each component's installation script and run them through pdsh, or use Apache Ambari (HMS): export namenode=nn1 export datanode=dn1,dn2,dn3,dn4 export secondary_namenode=sn1 export jobtracker=jt1 export tasktracker=tt1,tt2,tt3,tt4 export hbase_regionservers=rs1,rs2,rs3,rs4 export hbase_master=hbm # Setup Hadoop configuration /home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-conf.sh \ --java-home=/home/hms/apps/jdk1.6.0_27 \ --namenode-host=${namenode} \ --secondarynamenode-host=${secondary_namenode} \ --jobtracker-host=${jobtracker} \ --conf-dir=/etc/hadoop \ --hdfs-dir=/data/0/hadoop/var/hdfs,/data/1/hadoop/var/hdfs,/data/2/hadoop/var/hdfs,/data/3/hadoop/var/hdfs \ --namenode-dir=/data/2/hadoop/var/hdfs/name \ --mapred-dir=/data/0/tmp/mapred-local,/data/1/tmp/mapred-local,/data/2/tmp/mapred-local,/data/3/tmp/mapred-local \ --datanode-dir=/data/0/hadoop/var/hdfs/data,/data/1/hadoop/var/hdfs/data,/data/2/hadoop/var/hdfs/data,/data/3/hadoop/var/hdfs/data \ --log-dir=/home/hms/apps/hadoop-0.20.205.0/logs \ --pid-dir=/home/hms/apps/hadoop-0.20.205.0/logs \ --hdfs-user=hdfs \ --mapreduce-user=mr \ --kerberos-realm=APACHE.ORG \ --hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \ --mapreduce-user-keytab=/home/mr/mr.headless.keytab \ --datanodes=${datanode} \ --tasktrackers=${tasktracker} \ --group=hadoop \ --auto # Format Hadoop HDFS /home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-hdfs.sh \ --config /etc/hadoop \ --hdfs-user=hrt_hdfs \ --mapreduce-user=hrt_mr \ --kerberos-realm=APACHE.ORG \ --hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \ --format # Start up secondary name node, datanodes, job tracker, task tracker through hadoop-daemon.sh # Setup HBase /home/hms/apps/hbase-0.90.3/sbin/hbase-setup-conf.sh \ --hadoop-conf=/etc/hadoop \ --hadoop-home=/home/hms/apps/hadoop-0.20.205.0 \ --hadoop-namenode=${namenode} \ --hbase-home=/home/hms/apps/hbase-0.90.3 \ --hbase-conf=/home/hms/apps/hbase-0.90.3/conf \ --hbase-log=/home/hms/apps/hbase-0.90.3/logs \ --hbase-pid=/home/hms/apps/hbase-0.90.3/var/run \ --java-home=/usr/java/default \ --kerberos-realm=APACHE.ORG \ --kerberos-principal-id=hbase/_HOST \ --keytab-dir=/etc/security/keytabs \ --regionservers=${hbase_regionservers} \ --zookeeper-quorum=${hbase_master} \ --zookeeper-snapshot=/home/hms/apps/var/lib/zookeeper --hbase-user=hbase --hadoop-replication=3 # Start HBase Master, region servers using hbase-daemon.sh. Hope this helps.
          Hide
          Eric Yang added a comment -

          If no kerberos information is provided, the script will automatically setup as non-secure cluster.

          Show
          Eric Yang added a comment - If no kerberos information is provided, the script will automatically setup as non-secure cluster.
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-09-18 04:41:44, Lars Hofhansl wrote:

          > Generally it might be good to be able to define prefixes similar to what autoconf scripts do. Such as:

          > --prefix=DIR [PREFIX/etc]

          > --sysconfdir=DIR [PREFIX/etc]

          >

          > And so on.

          >

          > The hbase-site.xml and hbase-env.sh templates now have to be maintained separately, hence this makes config changes harder to maintain.

          >

          > In general this seems to be mostly useful for beginners of HBase, right?

          > (Oldtimers will have a mechanism in place to setup HBase - puppet, chef, fabric, etc.

          >

          > So I wonder if a more useful avenue would be to post this as part of the HBase book or some other getting started spot on the HBase site.

          >

          Hadoop related software used to deploy in isolated directory and use environment variables to drive the location of the software. GNU layout of single prefix does not work for tarball deployment because there are prefix directory per service. This is the reason that --hadoop-home, --hbase-home are implemented instead of --prefix.

          This is both useful for beginner or for script automation. I will include some documents for HBase book or HBase site.

          On 2011-09-18 04:41:44, Lars Hofhansl wrote:

          > /src/packages/hbase-setup-conf.sh, line 177

          > <https://reviews.apache.org/r/1925/diff/1/?file=41428#file41428line177>

          >

          > Isn't /usr a strange default for a Hadoop (or HBase or ZooKeeper) installation?

          > Maybe the script should rather check for a value, and fail if none was given.

          Existing tarball deployment does not have a default location, hence, the default is set to /usr for rpm/debian package default location. I will added a check for /usr/bin/hadoop and bail out if hadoop does not exist.

          On 2011-09-18 04:41:44, Lars Hofhansl wrote:

          > /src/packages/hbase-setup-conf.sh, line 245

          > <https://reviews.apache.org/r/1925/diff/1/?file=41428#file41428line245>

          >

          > I don't see HBASE_LOG_DIR defaulted anywhere.

          > Same for HBASE_PID_DIR.

          Good catch, I will make defaults for those two properties. How about default HBASE_LOG_DIR to $

          {HBASE_HOME}/var/log and ${HBASE_HOME}

          /var/run for consistency?

          On 2011-09-18 04:41:44, Lars Hofhansl wrote:

          > /src/packages/templates/conf/hbase-env.sh, line 3

          > <https://reviews.apache.org/r/1925/diff/1/?file=41429#file41429line3>

          >

          > Wow 2007... that's old

          > I think the latest convention is not to include he year.

          I will make the change to exclude the year. Thanks

          • Eric

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/#review1957
          -----------------------------------------------------------

          On 2011-09-16 00:06:21, Eric Yang wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1925/

          -----------------------------------------------------------

          (Updated 2011-09-16 00:06:21)

          Review request for hbase.

          Summary

          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:

          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location

          --hadoop-home=/usr Set Hadoop directory location

          --hadoop-namenode=localhost Set Hadoop namenode hostname

          --hadoop-replication=3 Set HDFS replication

          --hbase-home=/usr Set HBase directory location

          --hbase-conf=/etc/hbase Set HBase configuration directory location

          --hbase-log=/var/log/hbase Set HBase log directory location

          --hbase-pid=/var/run/hbase Set HBase pid directory location

          --hbase-user=hbase Set HBase user

          --java-home=/usr/java/default Set JAVA_HOME directory location

          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm

          --kerberos-principal-id=_HOST Set Kerberos principal ID

          --keytab-dir=/etc/security/keytabs Set keytab directory

          --regionservers=localhost Set regionservers hostnames

          --zookeeper-home=/usr Set ZooKeeper directory location

          --zookeeper-quorum=localhost Set ZooKeeper Quorum

          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.

          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs

          -----

          /src/assembly/all.xml 1170899

          /src/packages/hbase-setup-conf.sh PRE-CREATION

          /src/packages/templates/conf/hbase-env.sh PRE-CREATION

          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing

          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-09-18 04:41:44, Lars Hofhansl wrote: > Generally it might be good to be able to define prefixes similar to what autoconf scripts do. Such as: > --prefix=DIR [PREFIX/etc] > --sysconfdir=DIR [PREFIX/etc] > > And so on. > > The hbase-site.xml and hbase-env.sh templates now have to be maintained separately, hence this makes config changes harder to maintain. > > In general this seems to be mostly useful for beginners of HBase, right? > (Oldtimers will have a mechanism in place to setup HBase - puppet, chef, fabric, etc. > > So I wonder if a more useful avenue would be to post this as part of the HBase book or some other getting started spot on the HBase site. > Hadoop related software used to deploy in isolated directory and use environment variables to drive the location of the software. GNU layout of single prefix does not work for tarball deployment because there are prefix directory per service. This is the reason that --hadoop-home, --hbase-home are implemented instead of --prefix. This is both useful for beginner or for script automation. I will include some documents for HBase book or HBase site. On 2011-09-18 04:41:44, Lars Hofhansl wrote: > /src/packages/hbase-setup-conf.sh, line 177 > < https://reviews.apache.org/r/1925/diff/1/?file=41428#file41428line177 > > > Isn't /usr a strange default for a Hadoop (or HBase or ZooKeeper) installation? > Maybe the script should rather check for a value, and fail if none was given. Existing tarball deployment does not have a default location, hence, the default is set to /usr for rpm/debian package default location. I will added a check for /usr/bin/hadoop and bail out if hadoop does not exist. On 2011-09-18 04:41:44, Lars Hofhansl wrote: > /src/packages/hbase-setup-conf.sh, line 245 > < https://reviews.apache.org/r/1925/diff/1/?file=41428#file41428line245 > > > I don't see HBASE_LOG_DIR defaulted anywhere. > Same for HBASE_PID_DIR. Good catch, I will make defaults for those two properties. How about default HBASE_LOG_DIR to $ {HBASE_HOME}/var/log and ${HBASE_HOME} /var/run for consistency? On 2011-09-18 04:41:44, Lars Hofhansl wrote: > /src/packages/templates/conf/hbase-env.sh, line 3 > < https://reviews.apache.org/r/1925/diff/1/?file=41429#file41429line3 > > > Wow 2007... that's old > I think the latest convention is not to include he year. I will make the change to exclude the year. Thanks Eric ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/#review1957 ----------------------------------------------------------- On 2011-09-16 00:06:21, Eric Yang wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-16 00:06:21) Review request for hbase. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs ----- /src/assembly/all.xml 1170899 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          Eric Yang added a comment -
          • Added check for Hadoop installation.
          • Added default locations for HBase log/pid directories.
          • Added instruction to run hbase-setup-conf.sh in HBase Book.
          • Removed copyright date from license file.
          Show
          Eric Yang added a comment - Added check for Hadoop installation. Added default locations for HBase log/pid directories. Added instruction to run hbase-setup-conf.sh in HBase Book. Removed copyright date from license file.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/
          -----------------------------------------------------------

          (Updated 2011-09-20 01:17:07.234363)

          Review request for hbase.

          Changes
          -------

          • Added check for Hadoop installation.
          • Added default locations for HBase log/pid directories.
          • Added instruction to run hbase-setup-conf.sh in HBase Book.
          • Removed copyright date from license file.

          Summary
          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:
          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location
          --hadoop-home=/usr Set Hadoop directory location
          --hadoop-namenode=localhost Set Hadoop namenode hostname
          --hadoop-replication=3 Set HDFS replication
          --hbase-home=/usr Set HBase directory location
          --hbase-conf=/etc/hbase Set HBase configuration directory location
          --hbase-log=/var/log/hbase Set HBase log directory location
          --hbase-pid=/var/run/hbase Set HBase pid directory location
          --hbase-user=hbase Set HBase user
          --java-home=/usr/java/default Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
          --kerberos-principal-id=_HOST Set Kerberos principal ID
          --keytab-dir=/etc/security/keytabs Set keytab directory
          --regionservers=localhost Set regionservers hostnames
          --zookeeper-home=/usr Set ZooKeeper directory location
          --zookeeper-quorum=localhost Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.
          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs (updated)


          /src/assembly/all.xml 1170899
          /src/docbkx/getting_started.xml 1172936
          /src/packages/hbase-setup-conf.sh PRE-CREATION
          /src/packages/templates/conf/hbase-env.sh PRE-CREATION
          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing
          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-20 01:17:07.234363) Review request for hbase. Changes ------- Added check for Hadoop installation. Added default locations for HBase log/pid directories. Added instruction to run hbase-setup-conf.sh in HBase Book. Removed copyright date from license file. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs (updated) /src/assembly/all.xml 1170899 /src/docbkx/getting_started.xml 1172936 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/
          -----------------------------------------------------------

          (Updated 2011-09-20 01:20:32.140897)

          Review request for hbase.

          Changes
          -------

          Made a mistake in previous diff. Upload the same patch without specifying parent patch.

          • Added check for Hadoop installation.
          • Added default locations for HBase log/pid directories.
          • Added instruction to run hbase-setup-conf.sh in HBase Book.
          • Removed copyright date from license file.

          Summary
          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:
          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location
          --hadoop-home=/usr Set Hadoop directory location
          --hadoop-namenode=localhost Set Hadoop namenode hostname
          --hadoop-replication=3 Set HDFS replication
          --hbase-home=/usr Set HBase directory location
          --hbase-conf=/etc/hbase Set HBase configuration directory location
          --hbase-log=/var/log/hbase Set HBase log directory location
          --hbase-pid=/var/run/hbase Set HBase pid directory location
          --hbase-user=hbase Set HBase user
          --java-home=/usr/java/default Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
          --kerberos-principal-id=_HOST Set Kerberos principal ID
          --keytab-dir=/etc/security/keytabs Set keytab directory
          --regionservers=localhost Set regionservers hostnames
          --zookeeper-home=/usr Set ZooKeeper directory location
          --zookeeper-quorum=localhost Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.
          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs (updated)


          /src/assembly/all.xml 1172936
          /src/docbkx/getting_started.xml 1172936
          /src/packages/hbase-setup-conf.sh PRE-CREATION
          /src/packages/templates/conf/hbase-env.sh PRE-CREATION
          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing
          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-20 01:20:32.140897) Review request for hbase. Changes ------- Made a mistake in previous diff. Upload the same patch without specifying parent patch. Added check for Hadoop installation. Added default locations for HBase log/pid directories. Added instruction to run hbase-setup-conf.sh in HBase Book. Removed copyright date from license file. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs (updated) /src/assembly/all.xml 1172936 /src/docbkx/getting_started.xml 1172936 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          Giridharan Kesavan added a comment -

          can we have the option to set diff keytabs for hbasemaster and region servers?

          Show
          Giridharan Kesavan added a comment - can we have the option to set diff keytabs for hbasemaster and region servers?
          Hide
          Eric Yang added a comment -

          Updated the setup script to use unique key tabs for HBase master and region servers.

          Show
          Eric Yang added a comment - Updated the setup script to use unique key tabs for HBase master and region servers.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/
          -----------------------------------------------------------

          (Updated 2011-09-23 20:22:38.162933)

          Review request for hbase.

          Changes
          -------

          Updated the ability to use different keytab for HBase master and region server.

          Summary
          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:
          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location
          --hadoop-home=/usr Set Hadoop directory location
          --hadoop-namenode=localhost Set Hadoop namenode hostname
          --hadoop-replication=3 Set HDFS replication
          --hbase-home=/usr Set HBase directory location
          --hbase-conf=/etc/hbase Set HBase configuration directory location
          --hbase-log=/var/log/hbase Set HBase log directory location
          --hbase-pid=/var/run/hbase Set HBase pid directory location
          --hbase-user=hbase Set HBase user
          --java-home=/usr/java/default Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
          --kerberos-principal-id=_HOST Set Kerberos principal ID
          --keytab-dir=/etc/security/keytabs Set keytab directory
          --regionservers=localhost Set regionservers hostnames
          --zookeeper-home=/usr Set ZooKeeper directory location
          --zookeeper-quorum=localhost Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.
          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs (updated)


          /src/assembly/all.xml 1172936
          /src/docbkx/getting_started.xml 1172936
          /src/packages/hbase-setup-conf.sh PRE-CREATION
          /src/packages/templates/conf/hbase-env.sh PRE-CREATION
          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing
          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-23 20:22:38.162933) Review request for hbase. Changes ------- Updated the ability to use different keytab for HBase master and region server. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs (updated) /src/assembly/all.xml 1172936 /src/docbkx/getting_started.xml 1172936 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          Giridharan Kesavan added a comment -

          hbase-setup-conf.sh script check for $HADOOP_HOME/bin/hadoop and looks for $HADOOP_HOME/hadoop-core*.jar

          when HADOOP_HOME is set to /usr the first condition passes and the second condition fails for not being able to find the hadoop-core jar as the core jar is installed by hadoop rpm at /usr/share/hadoop/hadoop-core-*.jar

          Show
          Giridharan Kesavan added a comment - hbase-setup-conf.sh script check for $HADOOP_HOME/bin/hadoop and looks for $HADOOP_HOME/hadoop-core*.jar when HADOOP_HOME is set to /usr the first condition passes and the second condition fails for not being able to find the hadoop-core jar as the core jar is installed by hadoop rpm at /usr/share/hadoop/hadoop-core-*.jar
          Hide
          Eric Yang added a comment -

          Fixed location to find Hadoop jar file for both rpm and tarball deployment.

          Show
          Eric Yang added a comment - Fixed location to find Hadoop jar file for both rpm and tarball deployment.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1925/
          -----------------------------------------------------------

          (Updated 2011-09-23 23:17:00.742946)

          Review request for hbase.

          Changes
          -------

          Patch for finding location of hadoop jar files.

          Summary
          -------

          Create a post installation script to streamline configuration tasks for HBase.

          usage: /usr/sbin/hbase-setup-conf.sh <parameters>

          Optional parameters:
          --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location
          --hadoop-home=/usr Set Hadoop directory location
          --hadoop-namenode=localhost Set Hadoop namenode hostname
          --hadoop-replication=3 Set HDFS replication
          --hbase-home=/usr Set HBase directory location
          --hbase-conf=/etc/hbase Set HBase configuration directory location
          --hbase-log=/var/log/hbase Set HBase log directory location
          --hbase-pid=/var/run/hbase Set HBase pid directory location
          --hbase-user=hbase Set HBase user
          --java-home=/usr/java/default Set JAVA_HOME directory location
          --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
          --kerberos-principal-id=_HOST Set Kerberos principal ID
          --keytab-dir=/etc/security/keytabs Set keytab directory
          --regionservers=localhost Set regionservers hostnames
          --zookeeper-home=/usr Set ZooKeeper directory location
          --zookeeper-quorum=localhost Set ZooKeeper Quorum
          --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location

          This addresses bug HBASE-4415.
          https://issues.apache.org/jira/browse/HBASE-4415

          Diffs (updated)


          /src/assembly/all.xml 1175049
          /src/docbkx/getting_started.xml 1175049
          /src/packages/hbase-setup-conf.sh PRE-CREATION
          /src/packages/templates/conf/hbase-env.sh PRE-CREATION
          /src/packages/templates/conf/hbase-site.xml PRE-CREATION

          Diff: https://reviews.apache.org/r/1925/diff

          Testing
          -------

          Thanks,

          Eric

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1925/ ----------------------------------------------------------- (Updated 2011-09-23 23:17:00.742946) Review request for hbase. Changes ------- Patch for finding location of hadoop jar files. Summary ------- Create a post installation script to streamline configuration tasks for HBase. usage: /usr/sbin/hbase-setup-conf.sh <parameters> Optional parameters: --hadoop-conf=/etc/hadoop Set Hadoop configuration directory location --hadoop-home=/usr Set Hadoop directory location --hadoop-namenode=localhost Set Hadoop namenode hostname --hadoop-replication=3 Set HDFS replication --hbase-home=/usr Set HBase directory location --hbase-conf=/etc/hbase Set HBase configuration directory location --hbase-log=/var/log/hbase Set HBase log directory location --hbase-pid=/var/run/hbase Set HBase pid directory location --hbase-user=hbase Set HBase user --java-home=/usr/java/default Set JAVA_HOME directory location --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm --kerberos-principal-id=_HOST Set Kerberos principal ID --keytab-dir=/etc/security/keytabs Set keytab directory --regionservers=localhost Set regionservers hostnames --zookeeper-home=/usr Set ZooKeeper directory location --zookeeper-quorum=localhost Set ZooKeeper Quorum --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location This addresses bug HBASE-4415 . https://issues.apache.org/jira/browse/HBASE-4415 Diffs (updated) /src/assembly/all.xml 1175049 /src/docbkx/getting_started.xml 1175049 /src/packages/hbase-setup-conf.sh PRE-CREATION /src/packages/templates/conf/hbase-env.sh PRE-CREATION /src/packages/templates/conf/hbase-site.xml PRE-CREATION Diff: https://reviews.apache.org/r/1925/diff Testing ------- Thanks, Eric
          Hide
          Giridharan Kesavan added a comment -

          Eric,
          Could you please add the following to the hbaese-setup-conf script?
          set keytab directory permission to 700
          set the service keytab owner to hbaseuser

          Show
          Giridharan Kesavan added a comment - Eric, Could you please add the following to the hbaese-setup-conf script? set keytab directory permission to 700 set the service keytab owner to hbaseuser
          Hide
          Eric Yang added a comment -

          Added file permission setup for key tab files.

          Show
          Eric Yang added a comment - Added file permission setup for key tab files.
          Hide
          Eric Yang added a comment -

          Improve handling of chmod/chown if hbase master keytab doesn't exist in regional server.

          Show
          Eric Yang added a comment - Improve handling of chmod/chown if hbase master keytab doesn't exist in regional server.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12499070/HBASE-4415-6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499070/HBASE-4415-6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12499070/HBASE-4415-6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 8 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting
          org.apache.hadoop.hbase.regionserver.wal.TestLogRolling

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499070/HBASE-4415-6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 8 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Failed core tests, and javadoc warnings don't seem to be related to this patch.

          Show
          Eric Yang added a comment - Failed core tests, and javadoc warnings don't seem to be related to this patch.
          Hide
          Ted Yu added a comment -

          In update-hbase-env.sh:

          -. /etc/default/zookeeper-env.sh
          

          What's the rationale behind the above change ?

          Show
          Ted Yu added a comment - In update-hbase-env.sh: -. /etc/ default /zookeeper-env.sh What's the rationale behind the above change ?
          Hide
          Andrew Purtell added a comment -

          For 0.92, this patch is missing the as-yet undocumented security configuration settings. They are in HBASE-2742 and HBASE-3025 at the moment, and are expected to go in to a 0.92 RC.

          I refer to HBase specific security features, not the Hadoop security integration, which this patch seems to cover.

          Show
          Andrew Purtell added a comment - For 0.92, this patch is missing the as-yet undocumented security configuration settings. They are in HBASE-2742 and HBASE-3025 at the moment, and are expected to go in to a 0.92 RC. I refer to HBase specific security features, not the Hadoop security integration, which this patch seems to cover.
          Hide
          stack added a comment -

          Ditto on what Ted asks above.

          Why would we have this duplicated (and now behind) hbase-env.sh over in src/packages/templates/conf/hbase-env.sh? Why would we not copy it from original location?

          @Andrew I see this patch has:

          +  <property>
          +    <name>hbase.master.kerberos.principal</name>
          +    <value>${HBASE_M_K_PRINCIPAL}</value>
          +    <description></description>
          +  </property>
          

          We need more than that now?

          Show
          stack added a comment - Ditto on what Ted asks above. Why would we have this duplicated (and now behind) hbase-env.sh over in src/packages/templates/conf/hbase-env.sh? Why would we not copy it from original location? @Andrew I see this patch has: + <property> + <name>hbase.master.kerberos.principal</name> + <value>${HBASE_M_K_PRINCIPAL}</value> + <description></description> + </property> We need more than that now?
          Hide
          Eric Yang added a comment -

          Ted, HBASE-4498 contains the explanation of reason to remove zookeeper-env.sh because ZooKeeper community do not support this in their RPM creation.

          Stack, hbase-env.sh in src/packages/templates/conf is used for hbase-setup-conf.sh as post installation process. It is designed to be a template, where conf/hbase-env.sh can be edited in place, which I think they should be separated to be safe. I am happy to convert it to use conf/hbase-env.sh location, but this would make conf/hbase-env.sh a template. Is this something worth proceeding?

          Show
          Eric Yang added a comment - Ted, HBASE-4498 contains the explanation of reason to remove zookeeper-env.sh because ZooKeeper community do not support this in their RPM creation. Stack, hbase-env.sh in src/packages/templates/conf is used for hbase-setup-conf.sh as post installation process. It is designed to be a template, where conf/hbase-env.sh can be edited in place, which I think they should be separated to be safe. I am happy to convert it to use conf/hbase-env.sh location, but this would make conf/hbase-env.sh a template. Is this something worth proceeding?
          Hide
          stack added a comment -

          @Eric I don't want to bring this set of patches into 0.92. Though this patch was submitted a good while ago, its only gotten attention from us lot recently – thanks for keeping it up since original submit and sorry that you have had to kick us to take a look – but I think that there needs to be more review before it goes in (I can help get more reviews and I'll dig in more after I get an RC 0.92 out). Its more of a new feature than it is a critical bug fix so I'm fine w/ putting it off. We'll can get it into trunk soon. Good on you Eric.

          Show
          stack added a comment - @Eric I don't want to bring this set of patches into 0.92. Though this patch was submitted a good while ago, its only gotten attention from us lot recently – thanks for keeping it up since original submit and sorry that you have had to kick us to take a look – but I think that there needs to be more review before it goes in (I can help get more reviews and I'll dig in more after I get an RC 0.92 out). Its more of a new feature than it is a critical bug fix so I'm fine w/ putting it off. We'll can get it into trunk soon. Good on you Eric.
          Hide
          Eric Yang added a comment -

          Stack, no problem. If there is any change required, let me know.

          Show
          Eric Yang added a comment - Stack, no problem. If there is any change required, let me know.
          Hide
          Giridharan Kesavan added a comment -

          updating patch after HBASE-4894 commit.

          Show
          Giridharan Kesavan added a comment - updating patch after HBASE-4894 commit.
          Hide
          Arpit Gupta added a comment -

          now that HDFS-2246 (short circuit read) has been committed to hadoop, we need to support enabling of the short circuit read for the client. Attaching a new patch that also adds the support to set short circuit read configs.

          Show
          Arpit Gupta added a comment - now that HDFS-2246 (short circuit read) has been committed to hadoop, we need to support enabling of the short circuit read for the client. Attaching a new patch that also adds the support to set short circuit read configs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12505810/HBASE-4415-8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated -160 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAdmin
          org.apache.hadoop.hbase.client.TestInstantSchemaChange

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/419//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/419//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/419//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505810/HBASE-4415-8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestInstantSchemaChange Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/419//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/419//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/419//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12505818/HBASE-4415-9.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated -160 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAdmin
          org.apache.hadoop.hbase.client.TestInstantSchemaChange

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/420//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/420//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/420//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505818/HBASE-4415-9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestInstantSchemaChange Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/420//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/420//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/420//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Moving to 0.94.1 for now

          Show
          Lars Hofhansl added a comment - Moving to 0.94.1 for now

            People

            • Assignee:
              Unassigned
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development