HBase
  1. HBase
  2. HBASE-11604

Disable co-locating meta/master by default

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To avoid possible confusing, it's better to keep the original deployment scheme in 1.0. ZK-less region assignment is off by default in 1.0 already. We should, by default, not assign any region to master or backup master.

      1. hbase-11604.patch
        18 kB
        Jimmy Xiang
      2. hbase-11604_v2.patch
        21 kB
        Jimmy Xiang
      3. hbase-11604_v3.patch
        24 kB
        Jimmy Xiang
      4. hbase-11604_v3.1.patch
        27 kB
        Jimmy Xiang

        Issue Links

          Activity

          Hide
          Jimmy Xiang added a comment -

          Attached a patch for branch 1.

          Show
          Jimmy Xiang added a comment - Attached a patch for branch 1.
          Hide
          stack added a comment -

          +1

          A config that says 'regions on master' is confusing – regionservers only carry regions – but as I understand it transitional so I'm ok w/ this. Good stuff Jimmy Xiang

          Show
          stack added a comment - +1 A config that says 'regions on master' is confusing – regionservers only carry regions – but as I understand it transitional so I'm ok w/ this. Good stuff Jimmy Xiang
          Hide
          Jimmy Xiang added a comment -

          Cancelled the patch for now. Just realized that this change may impact the standalone/pseudo distributed mode. Let me think about it more.

          Show
          Jimmy Xiang added a comment - Cancelled the patch for now. Just realized that this change may impact the standalone/pseudo distributed mode. Let me think about it more.
          Hide
          Jimmy Xiang added a comment -

          Attached v2 with minor changes. Verified that standalone/pseudo distributed modes work fine.

          Show
          Jimmy Xiang added a comment - Attached v2 with minor changes. Verified that standalone/pseudo distributed modes work fine.
          Hide
          stack added a comment -

          Do we still need to specify two configurations? That we are distributed and many RS to check in or is that not needed anymore?

          Show
          stack added a comment - Do we still need to specify two configurations? That we are distributed and many RS to check in or is that not needed anymore?
          Hide
          Jimmy Xiang added a comment -

          For pseudo distributed mode, we still need to specify two configurations.

          Show
          Jimmy Xiang added a comment - For pseudo distributed mode, we still need to specify two configurations.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660722/hbase-11604_v2.patch
          against trunk revision .
          ATTACHMENT ID: 12660722

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10362//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660722/hbase-11604_v2.patch against trunk revision . ATTACHMENT ID: 12660722 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10362//console This message is automatically generated.
          Hide
          stack added a comment -

          For pseudo distributed mode, we still need to specify two configurations.

          Do you need two configs when in distributed mode too?

          Show
          stack added a comment - For pseudo distributed mode, we still need to specify two configurations. Do you need two configs when in distributed mode too?
          Hide
          Jimmy Xiang added a comment -

          For distributed mode, no, we don't. You just need to have region servers on other ports/machines.

          Show
          Jimmy Xiang added a comment - For distributed mode, no, we don't. You just need to have region servers on other ports/machines.
          Hide
          stack added a comment -

          If on one machine – i.e. pseudo distributed – can't we tell if the RS if ours or that of another RS? We know our startcode?

          I ask because asking for second config. just to do pseudo is going to be a little surprising and makes the setup of pseudo just a little bit more complicated. Thanks Jimmy.

          Show
          stack added a comment - If on one machine – i.e. pseudo distributed – can't we tell if the RS if ours or that of another RS? We know our startcode? I ask because asking for second config. just to do pseudo is going to be a little surprising and makes the setup of pseudo just a little bit more complicated. Thanks Jimmy.
          Hide
          Jimmy Xiang added a comment -

          The problem is that the default RPC port is occupied by the master so the RS can't start.

          Show
          Jimmy Xiang added a comment - The problem is that the default RPC port is occupied by the master so the RS can't start.
          Hide
          stack added a comment -

          The combined M+RS only puts up one port? And on this port is the master UI and RS UI? Or are we talking about the command ports, the ports we send protobuf over?

          Show
          stack added a comment - The combined M+RS only puts up one port? And on this port is the master UI and RS UI? Or are we talking about the command ports, the ports we send protobuf over?
          Hide
          Jimmy Xiang added a comment -

          The combined M+RS puts up three ports: one for RPC (original RS RPC port), one for the master and RS UI (original RS web UI port), and an extra that listens to the original master web UI port that is used for redirection.

          Show
          Jimmy Xiang added a comment - The combined M+RS puts up three ports: one for RPC (original RS RPC port), one for the master and RS UI (original RS web UI port), and an extra that listens to the original master web UI port that is used for redirection.
          Hide
          stack added a comment -

          If we by default are disabling colocating meta with master, what other regions would master be serving? Should the default be that it NOT serve regions? Would that help?

          Show
          stack added a comment - If we by default are disabling colocating meta with master, what other regions would master be serving? Should the default be that it NOT serve regions? Would that help?
          Hide
          Jimmy Xiang added a comment -

          That's right. master doesn't server regions by default.

          Show
          Jimmy Xiang added a comment - That's right. master doesn't server regions by default.
          Hide
          stack added a comment -

          So, which ports are clashing mighty Jibin? If M is not a RS by default, we should be good?

          Show
          stack added a comment - So, which ports are clashing mighty Jibin? If M is not a RS by default, we should be good?
          Hide
          Enis Soztutar added a comment -

          Agreed with Jimmy on disabling this by default. It gives a smoother migration story.

          Show
          Enis Soztutar added a comment - Agreed with Jimmy on disabling this by default. It gives a smoother migration story.
          Hide
          Jimmy Xiang added a comment -

          If M doesn't use the RS ports, we will be good.

          Show
          Jimmy Xiang added a comment - If M doesn't use the RS ports, we will be good.
          Hide
          stack added a comment -

          If M doesn't use the RS ports, we will be good.

          Is doing the above part of this patch or is this how it is now in branch-1? Thanks Jimmy.

          Show
          stack added a comment - If M doesn't use the RS ports, we will be good. Is doing the above part of this patch or is this how it is now in branch-1? Thanks Jimmy.
          Hide
          Jimmy Xiang added a comment -

          Neither. This patch changes the balancer so that by default no region is assigned to master.

          Show
          Jimmy Xiang added a comment - Neither. This patch changes the balancer so that by default no region is assigned to master.
          Hide
          stack added a comment -

          But ports clash because though the M doesn't have any regions, it is listening on the RS port? For pseudo distributed, we need to specify two configs still? What are they?

          Show
          stack added a comment - But ports clash because though the M doesn't have any regions, it is listening on the RS port? For pseudo distributed, we need to specify two configs still? What are they?
          Hide
          Enis Soztutar added a comment -

          Jimmy Xiang should we go ahead with this one. I can wait for this for 0.99.0 RC. wdyt?

          Show
          Enis Soztutar added a comment - Jimmy Xiang should we go ahead with this one. I can wait for this for 0.99.0 RC. wdyt?
          Hide
          Jimmy Xiang added a comment -

          But ports clash because though the M doesn't have any regions, it is listening on the RS port?

          That's right. If M doesn't listen on the RS port if it doesn't have any regions, will it cause more confusing?

          For pseudo distributed, we need to specify two configs still? What are they?

          Besides setting hbase.cluster.distributed to true, hbase.master.wait.on.regionservers.mintostart needs to be set to 1. I linked HBASE-11575 here, which has a release note about this change.

          should we go ahead with this one.

          This is ok with me. stack, what do you think?

          Show
          Jimmy Xiang added a comment - But ports clash because though the M doesn't have any regions, it is listening on the RS port? That's right. If M doesn't listen on the RS port if it doesn't have any regions, will it cause more confusing? For pseudo distributed, we need to specify two configs still? What are they? Besides setting hbase.cluster.distributed to true, hbase.master.wait.on.regionservers.mintostart needs to be set to 1. I linked HBASE-11575 here, which has a release note about this change. should we go ahead with this one. This is ok with me. stack , what do you think?
          Hide
          stack added a comment -

          I don't understand why we need to have user put in place two configs when previous we only had the user specify one when we are NOT colocating meta and master: i.e. the old behavior. To disable colocating meta and master we set a config? Can't we key off that and not start RS on M node when this is set.... and internally set hbase.master.wait.on.regionservers.mintostart to 1 w/o the user having to specify it?

          Show
          stack added a comment - I don't understand why we need to have user put in place two configs when previous we only had the user specify one when we are NOT colocating meta and master: i.e. the old behavior. To disable colocating meta and master we set a config? Can't we key off that and not start RS on M node when this is set.... and internally set hbase.master.wait.on.regionservers.mintostart to 1 w/o the user having to specify it?
          Hide
          Jimmy Xiang added a comment -

          Make sense. Let me fix this and update the refguid as well.

          Show
          Jimmy Xiang added a comment - Make sense. Let me fix this and update the refguid as well.
          Hide
          Jimmy Xiang added a comment -

          Attached v3 that set default hbase.master.wait.on.regionservers.mintostart to 1 if not configuring to put any region on master, otherwise set it to 2. Updated the refguide too. I verified now the pseudo mode can start without the extra configuration.

          Show
          Jimmy Xiang added a comment - Attached v3 that set default hbase.master.wait.on.regionservers.mintostart to 1 if not configuring to put any region on master, otherwise set it to 2. Updated the refguide too. I verified now the pseudo mode can start without the extra configuration.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667550/hbase-11604_v3.patch
          against trunk revision .
          ATTACHMENT ID: 12667550

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10803//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667550/hbase-11604_v3.patch against trunk revision . ATTACHMENT ID: 12667550 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10803//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          Jimmy Xiang do we want this patch only in branch-1?

          Show
          Enis Soztutar added a comment - Jimmy Xiang do we want this patch only in branch-1?
          Hide
          Jimmy Xiang added a comment -

          Yes, only for branch-1 for now.

          Show
          Jimmy Xiang added a comment - Yes, only for branch-1 for now.
          Hide
          Enis Soztutar added a comment -

          Ok, will commit this shortly. Patch looks good.

          Show
          Enis Soztutar added a comment - Ok, will commit this shortly. Patch looks good.
          Hide
          Jimmy Xiang added a comment -

          Attached v3.1 with minor changes to fix some flaky tests.

          Show
          Jimmy Xiang added a comment - Attached v3.1 with minor changes to fix some flaky tests.
          Hide
          stack added a comment -

          +1 on branch-1 only for now. What to do on master is up in the air still I'd say but we should be fixing that soon enough.

          Show
          stack added a comment - +1 on branch-1 only for now. What to do on master is up in the air still I'd say but we should be fixing that soon enough.
          Hide
          Jimmy Xiang added a comment -

          Integrated v3.1 to branch 1. Thanks everyone.

          Show
          Jimmy Xiang added a comment - Integrated v3.1 to branch 1. Thanks everyone.
          Hide
          Enis Soztutar added a comment -

          Yeah, was running unit tests before commit.

          org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer and org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster failed for me. Did you fix those in 3.1 patch?

          Show
          Enis Soztutar added a comment - Yeah, was running unit tests before commit. org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer and org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster failed for me. Did you fix those in 3.1 patch?
          Hide
          Jimmy Xiang added a comment -

          Yes, these two are fixed. Thanks.

          Show
          Jimmy Xiang added a comment - Yes, these two are fixed. Thanks.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-1.0 #177 (See https://builds.apache.org/job/HBase-1.0/177/)
          HBASE-11604 Disable co-locating meta/master by default (jxiang: rev d283818c51d8e69a77972c880369fdbd45b3297d)

          • src/main/docbkx/getting_started.xml
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenInitializing.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestZKLessAMOnCluster.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDistributedLogReplay.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterOperationsForRegionReplicas.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-1.0 #177 (See https://builds.apache.org/job/HBase-1.0/177/ ) HBASE-11604 Disable co-locating meta/master by default (jxiang: rev d283818c51d8e69a77972c880369fdbd45b3297d) src/main/docbkx/getting_started.xml hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenInitializing.java hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestZKLessAMOnCluster.java hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDistributedLogReplay.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterOperationsForRegionReplicas.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java
          Hide
          Enis Soztutar added a comment -

          Closing this issue after 0.99.0 release.

          Show
          Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

            People

            • Assignee:
              Jimmy Xiang
              Reporter:
              Jimmy Xiang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development