Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is a follow up to Async WAL replication jira (HBASE-11568). In HBASE-11568 we setup replication automatically in create table if the configuration is enabled.

      We should do the same in case a table with region replication = 1 is modified, and region replication is set to >1.

      1. hbase-13169_v1.patch
        4 kB
        Enis Soztutar
      2. hbase-13169_v2-master.patch
        3 kB
        Enis Soztutar

        Activity

        Hide
        enis Enis Soztutar added a comment -

        Attaching simple patch.

        Show
        enis Enis Soztutar added a comment - Attaching simple patch.
        Hide
        mantonov Mikhail Antonov added a comment -

        Looks good to me. Wondering if any special handling is needed if modification goes in "opposite" direction, say from 3 to 1 num of reps?

        Show
        mantonov Mikhail Antonov added a comment - Looks good to me. Wondering if any special handling is needed if modification goes in "opposite" direction, say from 3 to 1 num of reps?
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12703197/hbase-13169_v1.patch
        against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
        ATTACHMENT ID: 12703197

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703197/hbase-13169_v1.patch against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20. ATTACHMENT ID: 12703197 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13125//console This message is automatically generated.
        Hide
        jmhsieh Jonathan Hsieh added a comment -

        does this work for the online table modification case?

        nit: seems like the remove and add replicas would be good to have in one method, or pull the "ifNeeded" check into handleTableOperation to make it look parallel.

        Show
        jmhsieh Jonathan Hsieh added a comment - does this work for the online table modification case? nit: seems like the remove and add replicas would be good to have in one method, or pull the "ifNeeded" check into handleTableOperation to make it look parallel.
        Hide
        enis Enis Soztutar added a comment -

        Thanks for taking a look.

        Wondering if any special handling is needed if modification goes in "opposite" direction, say from 3 to 1 num of reps?

        We auto-add the replication peer when a table with region replicas is created or a table is modified to have replicas (with this patch). However, there is not yet support for auto-removing the replication peer if the last table with region replicas is deleted or replication is reduced to 1 via modify table. We do not have a global lock on all the table operations in master so doing an atomic check for "is this the last table with replicas" is hard to do.

        does this work for the online table modification case?

        Right now, online modifying a table for increasing region replicas is not supported (see ModifyTableHandler.prepareWithTableLock()).

        seems like the remove and add replicas would be good to have in one method, or pull the "ifNeeded" check into handleTableOperation to make it look parallel.

        I see your point. These two cases are doing different things (removing from meta vs adding a replication peer). Having separate methods should be good.

        Show
        enis Enis Soztutar added a comment - Thanks for taking a look. Wondering if any special handling is needed if modification goes in "opposite" direction, say from 3 to 1 num of reps? We auto-add the replication peer when a table with region replicas is created or a table is modified to have replicas (with this patch). However, there is not yet support for auto-removing the replication peer if the last table with region replicas is deleted or replication is reduced to 1 via modify table. We do not have a global lock on all the table operations in master so doing an atomic check for "is this the last table with replicas" is hard to do. does this work for the online table modification case? Right now, online modifying a table for increasing region replicas is not supported (see ModifyTableHandler.prepareWithTableLock()). seems like the remove and add replicas would be good to have in one method, or pull the "ifNeeded" check into handleTableOperation to make it look parallel. I see your point. These two cases are doing different things (removing from meta vs adding a replication peer). Having separate methods should be good.
        Hide
        mantonov Mikhail Antonov added a comment -

        We do not have a global lock on all the table operations in master so doing an atomic check for "is this the last table with replicas" is hard to do.

        I see, thanks. I guess we may have a chore on master watching for obsolete peers, but that'd be separate thing..

        Show
        mantonov Mikhail Antonov added a comment - We do not have a global lock on all the table operations in master so doing an atomic check for "is this the last table with replicas" is hard to do. I see, thanks. I guess we may have a chore on master watching for obsolete peers, but that'd be separate thing..
        Hide
        devaraj Devaraj Das added a comment -

        LGTM

        Show
        devaraj Devaraj Das added a comment - LGTM
        Hide
        enis Enis Soztutar added a comment - - edited

        I had committed this to master, but had to revert because the test seems flaky. Will look at that.

        Show
        enis Enis Soztutar added a comment - - edited I had committed this to master, but had to revert because the test seems flaky. Will look at that.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6228 (See https://builds.apache.org/job/HBase-TRUNK/6228/)
        HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE (enis: rev be25b0d595371e221da93a1b734dc696d044c9c8)

        • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6228 (See https://builds.apache.org/job/HBase-TRUNK/6228/ ) HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE (enis: rev be25b0d595371e221da93a1b734dc696d044c9c8) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6229 (See https://builds.apache.org/job/HBase-TRUNK/6229/)
        Revert "HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE" (enis: rev 21b27c865087580d75f08819144067308c4fd80c)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6229 (See https://builds.apache.org/job/HBase-TRUNK/6229/ ) Revert " HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE" (enis: rev 21b27c865087580d75f08819144067308c4fd80c) hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        Hide
        enis Enis Soztutar added a comment -

        Attaching v2 patch which should fix the flaky test. Run the test 20 times. Will commit if hadoopqa passes.

        Show
        enis Enis Soztutar added a comment - Attaching v2 patch which should fix the flaky test. Run the test 20 times. Will commit if hadoopqa passes.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704030/hbase-13169_v2-master.patch
        against master branch at commit c938267db3bfe95908f97397f64ffc50933e4343.
        ATTACHMENT ID: 12704030

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestImportExport
        org.apache.hadoop.hbase.util.TestProcessBasedCluster

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704030/hbase-13169_v2-master.patch against master branch at commit c938267db3bfe95908f97397f64ffc50933e4343. ATTACHMENT ID: 12704030 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportExport org.apache.hadoop.hbase.util.TestProcessBasedCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13201//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        I've committed v2 patch.

        Show
        enis Enis Soztutar added a comment - I've committed v2 patch.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6250 (See https://builds.apache.org/job/HBase-TRUNK/6250/)
        HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE (enis: rev d45efa72080e9b7ce77c69d1bd8e812e7ef1b698)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6250 (See https://builds.apache.org/job/HBase-TRUNK/6250/ ) HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE (enis: rev d45efa72080e9b7ce77c69d1bd8e812e7ef1b698) hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-1.1 #283 (See https://builds.apache.org/job/HBase-1.1/283/)
        HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE (enis: rev c8999d9defe9dd80822c26f0af40c7ed429519fc)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-1.1 #283 (See https://builds.apache.org/job/HBase-1.1/283/ ) HBASE-13169 ModifyTable increasing the region replica count should also auto-setup RRRE (enis: rev c8999d9defe9dd80822c26f0af40c7ed429519fc) hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Closing issues released in 1.1.0.

        Show
        ndimiduk Nick Dimiduk added a comment - Closing issues released in 1.1.0.

          People

          • Assignee:
            enis Enis Soztutar
            Reporter:
            enis Enis Soztutar
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development