HBase
  1. HBase
  2. HBASE-1511

Pseudo distributed mode in LocalHBaseCluster

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.19.3
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After changes to config and adding of parameter:

        <property>
          <name>hbase.cluster.distributed</name>
          <value>false</value>
          <description>The mode the cluster will be in. Possible values are
            false: standalone and pseudo-distributed setups with managed Zookeeper
            true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
          </description>
        </property>
      

      running HBase in pseudo destr mode become not so easy, I think description for false variant is wrong and you can't get pseudo-distributed mode setting this option to false now.

      I wish to have some option in LocalHBaseCluster for pseudo destr mode where I wish to have:

      • separate processes for Master and HRSs
      • ability to dynamically change HRS number in cluster
      • managed ZK cluster (like it is in local mode now)
      • managed small local HDFS cluster

      I think it is very useful for testing, debugging and for user experience with HBase, who want to try something close to real HBase not some emulation in standalone mode.

      1. psuedo-distribute hbase usage.txt
        2 kB
        Nicolas Spiegelberg
      2. HBASE-1511.patch
        12 kB
        Nicolas Spiegelberg

        Issue Links

          Activity

          Hide
          stack added a comment -

          Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.

          Show
          stack added a comment - Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.
          Hide
          Nicolas Spiegelberg added a comment -

          What I am tackling

          • separate processes for Master & RS
          • script to add more RS in a psuedo-distributed cluster
          • script to add backup masters in a psudo-distributed cluster
          • config file that
            a. explains basic psuedo-distributed mode
            b. shows the extra options to add for running on top of pseudo-distributed HDFS
          Show
          Nicolas Spiegelberg added a comment - What I am tackling separate processes for Master & RS script to add more RS in a psuedo-distributed cluster script to add backup masters in a psudo-distributed cluster config file that a. explains basic psuedo-distributed mode b. shows the extra options to add for running on top of pseudo-distributed HDFS
          Hide
          Nicolas Spiegelberg added a comment -

          psuedo-distributed ZK is more work than the others based upon the way that ZooKeeperWrapper & HQuorumPeer was written. Should still be doable, but noting some of the problems for consideration:

          1. Need to add conf item (hbase.zookeeper.myid) to override ID lookup in HQuorumPeer.writeMyID()
          2. should be different per HQuorumPeer: hbase.zookeeper.property.clientPort + hbase.zookeeper.property.dataDir
          3. hbase.zookeeper.property.server.<id> will be changed to <addr>:<peer_port>:<leader_port>:<client_port>
          a. We currently don't allow a custom peer_port + leader_port even though ZK supports it
          b. Need extra client_port for ZooKeeperWrapper.setQuorumServers()

          I have a feeling I'm going to be writing this code twice after Jonathan finishes the master rewrite.

          Show
          Nicolas Spiegelberg added a comment - psuedo-distributed ZK is more work than the others based upon the way that ZooKeeperWrapper & HQuorumPeer was written. Should still be doable, but noting some of the problems for consideration: 1. Need to add conf item (hbase.zookeeper.myid) to override ID lookup in HQuorumPeer.writeMyID() 2. should be different per HQuorumPeer: hbase.zookeeper.property.clientPort + hbase.zookeeper.property.dataDir 3. hbase.zookeeper.property.server.<id> will be changed to <addr>:<peer_port>:<leader_port>:<client_port> a. We currently don't allow a custom peer_port + leader_port even though ZK supports it b. Need extra client_port for ZooKeeperWrapper.setQuorumServers() I have a feeling I'm going to be writing this code twice after Jonathan finishes the master rewrite.
          Hide
          HBase Review Board added a comment -

          Message from: "Nicolas" <nspiegelberg@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/328/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          Pseudo-distributed mode for HBase

          • separate processes for Master & RS
          • script to add more RS in a psuedo-distributed cluster
          • script to add backup masters in a psudo-distributed cluster
          • config file that
            a. explains basic psuedo-distributed mode settings
            b. shows the extra options to add for running on top of pseudo-distributed HDFS

          Note that I needed to add a GNU Parsing utility to the master + regionserver so I could override certain config values on the fly instead of making seperate config files.

          This addresses bug HBASE-1511.
          http://issues.apache.org/jira/browse/HBASE-1511

          Diffs


          trunk/bin/local-master-backup.sh PRE-CREATION
          trunk/bin/local-regionservers.sh PRE-CREATION
          trunk/conf/hbase-site.xml.psuedo-distributed PRE-CREATION
          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 964686
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 964686

          Diff: http://review.hbase.org/r/328/diff

          Testing
          -------

          Thanks,

          Nicolas

          Show
          HBase Review Board added a comment - Message from: "Nicolas" <nspiegelberg@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/328/ ----------------------------------------------------------- Review request for hbase. Summary ------- Pseudo-distributed mode for HBase separate processes for Master & RS script to add more RS in a psuedo-distributed cluster script to add backup masters in a psudo-distributed cluster config file that a. explains basic psuedo-distributed mode settings b. shows the extra options to add for running on top of pseudo-distributed HDFS Note that I needed to add a GNU Parsing utility to the master + regionserver so I could override certain config values on the fly instead of making seperate config files. This addresses bug HBASE-1511 . http://issues.apache.org/jira/browse/HBASE-1511 Diffs trunk/bin/local-master-backup.sh PRE-CREATION trunk/bin/local-regionservers.sh PRE-CREATION trunk/conf/hbase-site.xml.psuedo-distributed PRE-CREATION trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 964686 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 964686 Diff: http://review.hbase.org/r/328/diff Testing ------- Thanks, Nicolas
          Hide
          HBase Review Board added a comment -

          Message from: stack@duboce.net

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/328/#review414
          -----------------------------------------------------------

          Ship it!

          This looks nice Nicolas. Want to take care of the few below and I'll then commit?

          trunk/bin/local-master-backup.sh
          <http://review.hbase.org/r/328/#comment1761>

          Why is it called local-master-backup? Should be a more descriptive name? Its the 'backup' that throws me off. 'local-masters'?

          trunk/bin/local-master-backup.sh
          <http://review.hbase.org/r/328/#comment1762>

          Explain 'offset' in this usage output? (Seems to be port offset from default?)

          trunk/bin/local-master-backup.sh
          <http://review.hbase.org/r/328/#comment1763>

          Isn't the master default port 60000, not 60020? 60020 is RS default port?

          trunk/conf/hbase-site.xml.psuedo-distributed
          <http://review.hbase.org/r/328/#comment1767>

          This is commented out because dfs.support.append doesn't work with local fs?

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1768>

          You are adding a tab here? Or white-space?

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1769>

          Add more comment in here... in particular on the '-splitting' suffix. Where's that come from?

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1770>

          Tab?

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1772>

          There is whitespace in here.

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1773>

          Sorry Nicolas... this looks like an old unused option. Should probably remove it.

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1774>

          This is a nice addition.

          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <http://review.hbase.org/r/328/#comment1775>

          White space

          • stack
          Show
          HBase Review Board added a comment - Message from: stack@duboce.net ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/328/#review414 ----------------------------------------------------------- Ship it! This looks nice Nicolas. Want to take care of the few below and I'll then commit? trunk/bin/local-master-backup.sh < http://review.hbase.org/r/328/#comment1761 > Why is it called local-master-backup? Should be a more descriptive name? Its the 'backup' that throws me off. 'local-masters'? trunk/bin/local-master-backup.sh < http://review.hbase.org/r/328/#comment1762 > Explain 'offset' in this usage output? (Seems to be port offset from default?) trunk/bin/local-master-backup.sh < http://review.hbase.org/r/328/#comment1763 > Isn't the master default port 60000, not 60020? 60020 is RS default port? trunk/conf/hbase-site.xml.psuedo-distributed < http://review.hbase.org/r/328/#comment1767 > This is commented out because dfs.support.append doesn't work with local fs? trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1768 > You are adding a tab here? Or white-space? trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1769 > Add more comment in here... in particular on the '-splitting' suffix. Where's that come from? trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1770 > Tab? trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1772 > There is whitespace in here. trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1773 > Sorry Nicolas... this looks like an old unused option. Should probably remove it. trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1774 > This is a nice addition. trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < http://review.hbase.org/r/328/#comment1775 > White space stack
          Hide
          HBase Review Board added a comment -

          Message from: "Jonathan Gray" <jgray@apache.org>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/328/#review417
          -----------------------------------------------------------

          trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <http://review.hbase.org/r/328/#comment1780>

          i think it's good to retain this option. once the new load balancer goes into place, you'll want it to not run the bulk placement to a small subset of the total servers during boot.

          could always add a maxWait timeout as well.

          • Jonathan
          Show
          HBase Review Board added a comment - Message from: "Jonathan Gray" <jgray@apache.org> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/328/#review417 ----------------------------------------------------------- trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java < http://review.hbase.org/r/328/#comment1780 > i think it's good to retain this option. once the new load balancer goes into place, you'll want it to not run the bulk placement to a small subset of the total servers during boot. could always add a maxWait timeout as well. Jonathan
          Hide
          HBase Review Board added a comment -

          Message from: "Nicolas" <nspiegelberg@facebook.com>

          On 2010-07-16 13:39:10, stack wrote:

          > This looks nice Nicolas. Want to take care of the few below and I'll then commit?

          I need to double-check to make sure that none of the ports will conflict at a low offset #. e.g.: shouldn't use 600+offset & 610+offset because it won't support more than 10 servers then. I think I saw that conflict last night, but just wanted to get some eyes/opinions on this.

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/bin/local-master-backup.sh, line 1

          > <http://review.hbase.org/r/328/diff/1/?file=2844#file2844line1>

          >

          > Why is it called local-master-backup? Should be a more descriptive name? Its the 'backup' that throws me off. 'local-masters'?

          >

          This JIRA also needs an associated wiki article change. So, the workflow would be:

          ./bin/start-hbase.sh
          ./bin/local-master-backup start 1

          I don't think you would want to use this file to start the original master, use start-hbase.sh for that. You primarily want to use this script to test master failover conditions.

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/bin/local-master-backup.sh, line 10

          > <http://review.hbase.org/r/328/diff/1/?file=2844#file2844line10>

          >

          > Explain 'offset' in this usage output? (Seems to be port offset from default?)

          offset is not only the port offset, but also an identifier for the logs. The log for "./bin/local-master-backup.sh start 1" would be located at logs/hbase-<user>-1-master-<server>.log

          In an internal review, Kannan asked why you couldn't use "start 0", I thought offset would help explain that issue. opinions?

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/bin/local-master-backup.sh, line 23

          > <http://review.hbase.org/r/328/diff/1/?file=2844#file2844line23>

          >

          > Isn't the master default port 60000, not 60020? 60020 is RS default port?

          oops. probably cut+paste error. can correct

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/conf/hbase-site.xml.psuedo-distributed, line 65

          > <http://review.hbase.org/r/328/diff/1/?file=2846#file2846line65>

          >

          > This is commented out because dfs.support.append doesn't work with local fs?

          exactly

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 637

          > <http://review.hbase.org/r/328/diff/1/?file=2847#file2847line637>

          >

          > You are adding a tab here? Or white-space?

          nasty editor! will fix

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 641

          > <http://review.hbase.org/r/328/diff/1/?file=2847#file2847line641>

          >

          > Add more comment in here... in particular on the '-splitting' suffix. Where's that come from?

          ooops. I had some stuff from HBASE-2312 sneak in. I'll correct that

          • Nicolas

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/328/#review414
          -----------------------------------------------------------

          Show
          HBase Review Board added a comment - Message from: "Nicolas" <nspiegelberg@facebook.com> On 2010-07-16 13:39:10, stack wrote: > This looks nice Nicolas. Want to take care of the few below and I'll then commit? I need to double-check to make sure that none of the ports will conflict at a low offset #. e.g.: shouldn't use 600+offset & 610+offset because it won't support more than 10 servers then. I think I saw that conflict last night, but just wanted to get some eyes/opinions on this. On 2010-07-16 13:39:10, stack wrote: > trunk/bin/local-master-backup.sh, line 1 > < http://review.hbase.org/r/328/diff/1/?file=2844#file2844line1 > > > Why is it called local-master-backup? Should be a more descriptive name? Its the 'backup' that throws me off. 'local-masters'? > This JIRA also needs an associated wiki article change. So, the workflow would be: ./bin/start-hbase.sh ./bin/local-master-backup start 1 I don't think you would want to use this file to start the original master, use start-hbase.sh for that. You primarily want to use this script to test master failover conditions. On 2010-07-16 13:39:10, stack wrote: > trunk/bin/local-master-backup.sh, line 10 > < http://review.hbase.org/r/328/diff/1/?file=2844#file2844line10 > > > Explain 'offset' in this usage output? (Seems to be port offset from default?) offset is not only the port offset, but also an identifier for the logs. The log for "./bin/local-master-backup.sh start 1" would be located at logs/hbase-<user>-1-master-<server>.log In an internal review, Kannan asked why you couldn't use "start 0", I thought offset would help explain that issue. opinions? On 2010-07-16 13:39:10, stack wrote: > trunk/bin/local-master-backup.sh, line 23 > < http://review.hbase.org/r/328/diff/1/?file=2844#file2844line23 > > > Isn't the master default port 60000, not 60020? 60020 is RS default port? oops. probably cut+paste error. can correct On 2010-07-16 13:39:10, stack wrote: > trunk/conf/hbase-site.xml.psuedo-distributed, line 65 > < http://review.hbase.org/r/328/diff/1/?file=2846#file2846line65 > > > This is commented out because dfs.support.append doesn't work with local fs? exactly On 2010-07-16 13:39:10, stack wrote: > trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 637 > < http://review.hbase.org/r/328/diff/1/?file=2847#file2847line637 > > > You are adding a tab here? Or white-space? nasty editor! will fix On 2010-07-16 13:39:10, stack wrote: > trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 641 > < http://review.hbase.org/r/328/diff/1/?file=2847#file2847line641 > > > Add more comment in here... in particular on the '-splitting' suffix. Where's that come from? ooops. I had some stuff from HBASE-2312 sneak in. I'll correct that Nicolas ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/328/#review414 -----------------------------------------------------------
          Hide
          Nicolas Spiegelberg added a comment -

          Updated patch to address Review Board comments. Also note:

          local-regionservers.sh : changed ports from 60020/60030 to 60200/60300. this is to accommodate up to 1000 regionservers/machine. kept master ports at 60000/60010 because I assumed you would not realistically want more than 10 masters. limitations are noted in the header comments

          Show
          Nicolas Spiegelberg added a comment - Updated patch to address Review Board comments. Also note: local-regionservers.sh : changed ports from 60020/60030 to 60200/60300. this is to accommodate up to 1000 regionservers/machine. kept master ports at 60000/60010 because I assumed you would not realistically want more than 10 masters. limitations are noted in the header comments
          Hide
          Nicolas Spiegelberg added a comment -

          Note that the executable state of local-master-backup.sh & local-regionserver.sh seems to have been lost in the maze of svn diff / patch -p0. Those two files need: svn ps svn:executable

          Show
          Nicolas Spiegelberg added a comment - Note that the executable state of local-master-backup.sh & local-regionserver.sh seems to have been lost in the maze of svn diff / patch -p0. Those two files need: svn ps svn:executable
          Hide
          Nicolas Spiegelberg added a comment -

          A simple usage document to get started on creating an associated wiki article

          Show
          Nicolas Spiegelberg added a comment - A simple usage document to get started on creating an associated wiki article
          Hide
          stack added a comment -

          Thanks for the sweet functionality Nicolas. Committed.

          Show
          stack added a comment - Thanks for the sweet functionality Nicolas. Committed.
          Hide
          stack added a comment -

          I converted your doc into an article that now shows in the left hand navbar when we build doc.

          Show
          stack added a comment - I converted your doc into an article that now shows in the left hand navbar when we build doc.
          Hide
          HBase Review Board added a comment -

          Message from: stack@duboce.net

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/bin/local-master-backup.sh, line 1

          > <http://review.cloudera.org/r/328/diff/1/?file=2844#file2844line1>

          >

          > Why is it called local-master-backup? Should be a more descriptive name? Its the 'backup' that throws me off. 'local-masters'?

          >

          Nicolas wrote:

          This JIRA also needs an associated wiki article change. So, the workflow would be:

          ./bin/start-hbase.sh

          ./bin/local-master-backup start 1

          I don't think you would want to use this file to start the original master, use start-hbase.sh for that. You primarily want to use this script to test master failover conditions.

          Don't write a wiki article. Add an article under src/site, under xdoc or apt (not yet in place). This can be done in a separate issue. Meantime, on commit I can add the above on how its used (w/o it, I was a little baffled but just those few lines above explain it all – for me at least).

          On 2010-07-16 13:39:10, stack wrote:

          > trunk/bin/local-master-backup.sh, line 10

          > <http://review.cloudera.org/r/328/diff/1/?file=2844#file2844line10>

          >

          > Explain 'offset' in this usage output? (Seems to be port offset from default?)

          Nicolas wrote:

          offset is not only the port offset, but also an identifier for the logs. The log for "./bin/local-master-backup.sh start 1" would be located at logs/hbase-<user>-1-master-<server>.log

          In an internal review, Kannan asked why you couldn't use "start 0", I thought offset would help explain that issue. opinions?

          Hmm... maybe you do need an article on how this stuff works. Smile. Offset is fine (Is Kannan saying that a 'start 0' should be rejected?). If you put together a paragraph or two, I'll take care of sticking it into proper format.

          • stack

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/328/#review414
          -----------------------------------------------------------

          Show
          HBase Review Board added a comment - Message from: stack@duboce.net On 2010-07-16 13:39:10, stack wrote: > trunk/bin/local-master-backup.sh, line 1 > < http://review.cloudera.org/r/328/diff/1/?file=2844#file2844line1 > > > Why is it called local-master-backup? Should be a more descriptive name? Its the 'backup' that throws me off. 'local-masters'? > Nicolas wrote: This JIRA also needs an associated wiki article change. So, the workflow would be: ./bin/start-hbase.sh ./bin/local-master-backup start 1 I don't think you would want to use this file to start the original master, use start-hbase.sh for that. You primarily want to use this script to test master failover conditions. Don't write a wiki article. Add an article under src/site, under xdoc or apt (not yet in place). This can be done in a separate issue. Meantime, on commit I can add the above on how its used (w/o it, I was a little baffled but just those few lines above explain it all – for me at least). On 2010-07-16 13:39:10, stack wrote: > trunk/bin/local-master-backup.sh, line 10 > < http://review.cloudera.org/r/328/diff/1/?file=2844#file2844line10 > > > Explain 'offset' in this usage output? (Seems to be port offset from default?) Nicolas wrote: offset is not only the port offset, but also an identifier for the logs. The log for "./bin/local-master-backup.sh start 1" would be located at logs/hbase-<user>-1-master-<server>.log In an internal review, Kannan asked why you couldn't use "start 0", I thought offset would help explain that issue. opinions? Hmm... maybe you do need an article on how this stuff works. Smile. Offset is fine (Is Kannan saying that a 'start 0' should be rejected?). If you put together a paragraph or two, I'll take care of sticking it into proper format. stack ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/328/#review414 -----------------------------------------------------------

            People

            • Assignee:
              Nicolas Spiegelberg
              Reporter:
              Evgeny Ryabitskiy
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development