HBase
  1. HBase
  2. HBASE-9863

Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs in admin#createTable() call

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry sometimes hung.
      Here were two recent occurrences:
      https://builds.apache.org/job/PreCommit-HBASE-Build/7676/console
      https://builds.apache.org/job/PreCommit-HBASE-Build/7671/console

      There were 9 occurrences of the following in both stack traces:

      "FifoRpcScheduler.handler1-thread-5" daemon prio=10 tid=0x09df8800 nid=0xc17 waiting for monitor entry [0x6fdf8000]
         java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
        - waiting to lock <0x7f69b5f0> (a org.apache.hadoop.hbase.master.TableNamespaceManager)
        at org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3146)
        at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3105)
        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1743)
        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1782)
        at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38221)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1983)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
      

      The test hung here:

      "pool-1-thread-1" prio=10 tid=0x74f7b800 nid=0x5aa5 in Object.wait() [0x74efe000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xcc848348> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
        at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1436)
        - locked <0xcc848348> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
        at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
        at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
        at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.createTable(MasterProtos.java:40372)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.createTable(HConnectionManager.java:1931)
        at org.apache.hadoop.hbase.client.HBaseAdmin$2.call(HBaseAdmin.java:598)
        at org.apache.hadoop.hbase.client.HBaseAdmin$2.call(HBaseAdmin.java:594)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
        - locked <0x7faa26d0> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
        - locked <0x7faa26d0> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3124)
        at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:594)
        at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:485)
        at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)
      
      1. 9863-v1.txt
        0.6 kB
        Ted Yu
      2. 9863-v2.txt
        0.7 kB
        Ted Yu
      3. 9863-v3.txt
        3 kB
        Ted Yu
      4. 9863-v4.txt
        4 kB
        Ted Yu
      5. 9863-v5.txt
        3 kB
        Ted Yu
      6. 9863-v6.txt
        3 kB
        Ted Yu

        Activity

        Hide
        Andrew Purtell added a comment -

        Already committed, resolving.

        Show
        Andrew Purtell added a comment - Already committed, resolving.
        Hide
        Ted Yu added a comment -

        TestZooKeeper hasn't failed lately.

        Show
        Ted Yu added a comment - TestZooKeeper hasn't failed lately.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4669 (See https://builds.apache.org/job/HBase-TRUNK/4669/)
        HBASE-9863 Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs (tedyu: rev 1539129)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4669 (See https://builds.apache.org/job/HBase-TRUNK/4669/ ) HBASE-9863 Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs (tedyu: rev 1539129) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #827 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/827/)
        HBASE-9863 Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs (tedyu: rev 1539129)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #827 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/827/ ) HBASE-9863 Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs (tedyu: rev 1539129) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
        Hide
        Ted Yu added a comment -

        Integrated to trunk.

        Thanks for the reviews.

        Show
        Ted Yu added a comment - Integrated to trunk. Thanks for the reviews.
        Hide
        Ted Yu added a comment -

        TestHRegionBusyWait failure was not related to patch.

        Planning to integrate patch v6 tomorrow if there is no further review comment.

        Show
        Ted Yu added a comment - TestHRegionBusyWait failure was not related to patch. Planning to integrate patch v6 tomorrow if there is no further review comment.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612034/9863-v6.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612034/9863-v6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7728//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch v6 adopts Jimmy's latest suggestion.

        Show
        Ted Yu added a comment - Patch v6 adopts Jimmy's latest suggestion.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12611985/9863-v5.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
        org.apache.hadoop.hbase.regionserver.TestHRegion

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611985/9863-v5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait org.apache.hadoop.hbase.regionserver.TestHRegion Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7727//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Now that tableNamespaceManager#isTableAvailableAndInitialized() becomes private, HMaster doesn't have access to it.
        Should I make that method public so that HMaster can check ?

        Show
        Ted Yu added a comment - Now that tableNamespaceManager#isTableAvailableAndInitialized() becomes private, HMaster doesn't have access to it. Should I make that method public so that HMaster can check ?
        Hide
        Ted Yu added a comment -

        isTableNamespaceManagerReady() is gone in HMaster.
        Originally only 'NamespaceDescriptor getNamespaceDescriptor(String name)' throws exception. I don't want to throw exception from TableNamespaceManager#get() which would change the behavior.

        Patch v6 coming.

        Show
        Ted Yu added a comment - isTableNamespaceManagerReady() is gone in HMaster. Originally only 'NamespaceDescriptor getNamespaceDescriptor(String name)' throws exception. I don't want to throw exception from TableNamespaceManager#get() which would change the behavior. Patch v6 coming.
        Hide
        Jimmy Xiang added a comment -

        For the get call, if isTableAvailableAndInitialized() returns false, originally, we throw an exception like

        -      throw new IOException("Table Namespace Manager not ready yet, try again later");
        

        Should we do the same? You can fix it before commit. +1.

        Show
        Jimmy Xiang added a comment - For the get call, if isTableAvailableAndInitialized() returns false, originally, we throw an exception like - throw new IOException( "Table Namespace Manager not ready yet, try again later" ); Should we do the same? You can fix it before commit. +1.
        Hide
        Ted Yu added a comment -

        Patch v5, based on v3, adds isTableAvailableAndInitialized() check in 'NamespaceDescriptor get(String name)'

        Show
        Ted Yu added a comment - Patch v5, based on v3, adds isTableAvailableAndInitialized() check in 'NamespaceDescriptor get(String name)'
        Hide
        Ted Yu added a comment - - edited

        For patch v4, the following findbug warning was introduced:

        SWL org.apache.hadoop.hbase.master.TableNamespaceManager.start() calls Thread.sleep() with a lock held

        Show
        Ted Yu added a comment - - edited For patch v4, the following findbug warning was introduced: SWL org.apache.hadoop.hbase.master.TableNamespaceManager.start() calls Thread.sleep() with a lock held
        Hide
        Ted Yu added a comment - - edited

        Same test failure (TestHRegionXX) appeared in QA run for HBASE-9808
        I don't think it was caused by my patch.

        Show
        Ted Yu added a comment - - edited Same test failure (TestHRegionXX) appeared in QA run for HBASE-9808 I don't think it was caused by my patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12611850/9863-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestHRegion
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611850/9863-v4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7721//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch v4 incorporates Jimmy's comments above.

        Show
        Ted Yu added a comment - Patch v4 incorporates Jimmy's comments above.
        Hide
        Jimmy Xiang added a comment -

        I think start() should be synchronized too. It's better to make sure it won't be called more than once also. For 'NamespaceDescriptor get(String name)', yes, the semantics is the same. After the change, we don't check if isTableAvailableAndInitialized. So zkNamespaceManager could be uninitialized.

        Show
        Jimmy Xiang added a comment - I think start() should be synchronized too. It's better to make sure it won't be called more than once also. For 'NamespaceDescriptor get(String name)', yes, the semantics is the same. After the change, we don't check if isTableAvailableAndInitialized. So zkNamespaceManager could be uninitialized.
        Hide
        Ted Yu added a comment -

        can we remove synchronized, instead make sure the callers have proper synchronization?

        isTableAvailableAndInitialized() is called by start() which is not synchronized. Doing the above would equate to adding synchronized(this) in start() before calling isTableAvailableAndInitialized().
        Is that what you meant ?

        For 'NamespaceDescriptor get(String name)', the semantics is the same as the original method.

        Show
        Ted Yu added a comment - can we remove synchronized, instead make sure the callers have proper synchronization? isTableAvailableAndInitialized() is called by start() which is not synchronized. Doing the above would equate to adding synchronized(this) in start() before calling isTableAvailableAndInitialized(). Is that what you meant ? For 'NamespaceDescriptor get(String name)', the semantics is the same as the original method.
        Hide
        Jimmy Xiang added a comment -

        For those private methods such as isTableAvailableAndInitialized() and getNamespaceTable(), can we remove synchronized, instead make sure the callers have proper synchronization? For this one:

           public synchronized NamespaceDescriptor get(String name) throws IOException {
        -    return get(getNamespaceTable(), name);
        +    return zkNamespaceManager.get(name);
           }
        

        The change is good. We still need to check is the manger is initialized, right?

        Show
        Jimmy Xiang added a comment - For those private methods such as isTableAvailableAndInitialized() and getNamespaceTable(), can we remove synchronized, instead make sure the callers have proper synchronization? For this one: public synchronized NamespaceDescriptor get( String name) throws IOException { - return get(getNamespaceTable(), name); + return zkNamespaceManager.get(name); } The change is good. We still need to check is the manger is initialized, right?
        Hide
        stack added a comment -

        You ok reviewing this last patch Jimmy Xiang? I am afraid to. Every time I ask a question, a new patch is posted which makes me ask another question...

        Show
        stack added a comment - You ok reviewing this last patch Jimmy Xiang ? I am afraid to. Every time I ask a question, a new patch is posted which makes me ask another question...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12611786/9863-v3.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611786/9863-v3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7715//console This message is automatically generated.
        Hide
        Ted Yu added a comment -
        Show
        Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/7712//testReport/org.apache.hadoop.hbase.security.access/TestNamespaceCommands/org_apache_hadoop_hbase_security_access_TestNamespaceCommands/ , it was not clear why hbase:acl table didn't come online. I wasn't able to reproduce the failure locally.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12611777/9863-v3.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.security.access.TestNamespaceCommands

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611777/9863-v3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.security.access.TestNamespaceCommands Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7712//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch v3, based on v1, adopts Jimmy's suggestion.

        Show
        Ted Yu added a comment - Patch v3, based on v1, adopts Jimmy's suggestion.
        Hide
        Jimmy Xiang added a comment -

        I have a suggestion:
        1. change TableNamespaceManager#public synchronized boolean isTableAvailableAndInitialized() to private boolean isTableAvailableAndInitialized();
        2. remove HMaster#isTableNamespaceManagerReady since we check isTableAvailableAndInitialized for each call inside TableNamespaceManager.

        As to the test failure, it could be because the test stuck while it tries to read the ns table.

        Show
        Jimmy Xiang added a comment - I have a suggestion: 1. change TableNamespaceManager#public synchronized boolean isTableAvailableAndInitialized() to private boolean isTableAvailableAndInitialized(); 2. remove HMaster#isTableNamespaceManagerReady since we check isTableAvailableAndInitialized for each call inside TableNamespaceManager. As to the test failure, it could be because the test stuck while it tries to read the ns table.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12611703/9863-v2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611703/9863-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7706//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        It would be larger refactoring so that access to zkNamespaceManager uses its own lock.
        Suggest doing that work in separate JIRA.

        At the moment, patch v1 looks better.

        Show
        Ted Yu added a comment - It would be larger refactoring so that access to zkNamespaceManager uses its own lock. Suggest doing that work in separate JIRA. At the moment, patch v1 looks better.
        Hide
        stack added a comment -

        Is that safe?

        Show
        stack added a comment - Is that safe?
        Hide
        Ted Yu added a comment -

        Patch v2 removes 'synchronized' from the get method.

        Test suite passed based on hadoop-2 profile on Linux.

        Show
        Ted Yu added a comment - Patch v2 removes 'synchronized' from the get method. Test suite passed based on hadoop-2 profile on Linux.
        Hide
        Ted Yu added a comment -

        Looking at zkNamespaceManager#get(), it calls ConcurrentSkipListMap#get() directly and returns.

        One thing we can do is to remove 'synchronized' from TableNamespaceManager#get(String name) method.

        Show
        Ted Yu added a comment - Looking at zkNamespaceManager#get(), it calls ConcurrentSkipListMap#get() directly and returns. One thing we can do is to remove 'synchronized' from TableNamespaceManager#get(String name) method.
        Hide
        stack added a comment -

        Just to say I don't mind digging in deeper since this is fix to a flakey unit tests and fixing these is important, hard work.

        Show
        stack added a comment - Just to say I don't mind digging in deeper since this is fix to a flakey unit tests and fixing these is important, hard work.
        Hide
        stack added a comment -

        I am just responding to what you wrote:

        This change reduces the period of time where lock on TableNamespaceManager instance is held.

        Now you are thinking that there should be no locking problem? But you end with:

        There shouldn't be hanging anymore.

        ... so maybe there will be locking?

        If you were skirting the locking with this change, I'd +1 this patch. But maybe I should dig in deeper with a review?

        Show
        stack added a comment - I am just responding to what you wrote: This change reduces the period of time where lock on TableNamespaceManager instance is held. Now you are thinking that there should be no locking problem? But you end with: There shouldn't be hanging anymore. ... so maybe there will be locking? If you were skirting the locking with this change, I'd +1 this patch. But maybe I should dig in deeper with a review?
        Hide
        Ted Yu added a comment -

        we could still hang?

        From HMaster:

          public NamespaceDescriptor getNamespaceDescriptor(String name) throws IOException {
            if (!isTableNamespaceManagerReady()) {
              throw new IOException("Table Namespace Manager not ready yet, try again later");
            }
            NamespaceDescriptor nsd = tableNamespaceManager.get(name);
        

        When tableNamespaceManager.get() is called, tableNamespaceManager is ready (initialized). zkNamespaceManager is able to serve request.
        There shouldn't be hanging anymore.

        Show
        Ted Yu added a comment - we could still hang? From HMaster: public NamespaceDescriptor getNamespaceDescriptor( String name) throws IOException { if (!isTableNamespaceManagerReady()) { throw new IOException( "Table Namespace Manager not ready yet, try again later" ); } NamespaceDescriptor nsd = tableNamespaceManager.get(name); When tableNamespaceManager.get() is called, tableNamespaceManager is ready (initialized). zkNamespaceManager is able to serve request. There shouldn't be hanging anymore.
        Hide
        stack added a comment -

        So we could still hang? The change just makes it less likely? Can we factor the way the locking is done here?

        Show
        stack added a comment - So we could still hang? The change just makes it less likely? Can we factor the way the locking is done here?
        Hide
        Ted Yu added a comment -

        The get(HTable table, String name) call goes through HTable.get() - an RPC.
        The patch replaces the above call with zkNamespaceManager.get(name) where zkNamespaceManager utilizes its cache.

        This change reduces the period of time where lock on TableNamespaceManager instance is held.
        I ran the patch through test suite on Linux using hadoop-2 profile and the suite passed.

        Show
        Ted Yu added a comment - The get(HTable table, String name) call goes through HTable.get() - an RPC. The patch replaces the above call with zkNamespaceManager.get(name) where zkNamespaceManager utilizes its cache. This change reduces the period of time where lock on TableNamespaceManager instance is held. I ran the patch through test suite on Linux using hadoop-2 profile and the suite passed.
        Hide
        stack added a comment -

        Sounds good. How does the change address the described problem? Thanks.

        Show
        stack added a comment - Sounds good. How does the change address the described problem? Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12611589/9863-v1.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611589/9863-v1.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7702//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        object monitor was held by the following call:

        "FifoRpcScheduler.handler1-thread-4" daemon prio=10 tid=0x09479800 nid=0x2b48 waiting on condition [0x6f157000]
           java.lang.Thread.State: TIMED_WAITING (sleeping)
          at java.lang.Thread.sleep(Native Method)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:148)
          - locked <0xd399b5a8> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
          at org.apache.hadoop.hbase.client.HTable.get(HTable.java:753)
          at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:134)
          at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:118)
          - locked <0x7f7828f0> (a org.apache.hadoop.hbase.master.TableNamespaceManager)
        

        Many other threads were blocked on 0x7f7828f0 :

        "FifoRpcScheduler.handler1-thread-5" daemon prio=10 tid=0x71ba6400 nid=0x35a7 waiting for monitor entry [0x6ef20000]
           java.lang.Thread.State: BLOCKED (on object monitor)
          at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
          - waiting to lock <0x7f7828f0> (a org.apache.hadoop.hbase.master.TableNamespaceManager)
          at org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3146)
          at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3105)
        
        Show
        Ted Yu added a comment - object monitor was held by the following call: "FifoRpcScheduler.handler1-thread-4" daemon prio=10 tid=0x09479800 nid=0x2b48 waiting on condition [0x6f157000] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:148) - locked <0xd399b5a8> (a org.apache.hadoop.hbase.client.RpcRetryingCaller) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:753) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:134) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:118) - locked <0x7f7828f0> (a org.apache.hadoop.hbase.master.TableNamespaceManager) Many other threads were blocked on 0x7f7828f0 : "FifoRpcScheduler.handler1-thread-5" daemon prio=10 tid=0x71ba6400 nid=0x35a7 waiting for monitor entry [0x6ef20000] java.lang. Thread .State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250) - waiting to lock <0x7f7828f0> (a org.apache.hadoop.hbase.master.TableNamespaceManager) at org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3146) at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3105)

          People

          • Assignee:
            Ted Yu
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development