HBase
  1. HBase
  2. HBASE-5153

Add retry logic in HConnectionImplementation#resetZooKeeperTrackers

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.4
    • Fix Version/s: 0.90.6
    • Component/s: Client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBASE-4893 is related to this issue. In that issue, we know, if multi-threads share a same connection, once this connection got abort in one thread, the other threads will got a "HConnectionManager$HConnectionImplementation@18fb1f7 closed" exception.

      It solve the problem of "stale connection can't removed". But the orignal HTable instance cann't be continue to use. The connection in HTable should be recreated.

      Actually, there's two aproach to solve this:
      1. In user code, once catch an IOE, close connection and re-create HTable instance. We can use this as a workaround.
      2. In HBase Client side, catch this exception, and re-create connection.

      1. TestResults-hbase5153.out
        26 kB
        Jieshan Bean
      2. HBASE-5153-V6-90-minorchange.patch
        27 kB
        Jieshan Bean
      3. HBASE-5153-V6-90.txt
        26 kB
        Jieshan Bean
      4. HBASE-5153-V5-90.patch
        26 kB
        Jieshan Bean
      5. HBASE-5153-V4-90.patch
        26 kB
        Jieshan Bean
      6. HBASE-5153-V3.patch
        19 kB
        Jieshan Bean
      7. HBASE-5153-V2.patch
        6 kB
        Jieshan Bean
      8. HBASE-5153-trunk-v2.patch
        12 kB
        Jieshan Bean
      9. HBASE-5153-trunk.patch
        12 kB
        Jieshan Bean
      10. HBase-5153-90-addendum.patch
        11 kB
        Jieshan Bean
      11. HBASE-5153.patch
        4 kB
        Jieshan Bean
      12. HBASE-5153_addendum_0.90_1.patch
        11 kB
        ramkrishna.s.vasudevan
      13. 5153-trunk-v2.txt
        13 kB
        Lars Hofhansl
      14. 5153-trunk-minimal.txt
        6 kB
        Lars Hofhansl
      15. 5153-trunk.txt
        12 kB
        Ted Yu
      16. 5153-trunk.txt
        26 kB
        Ted Yu
      17. 5153-92.txt
        25 kB
        Ted Yu

        Issue Links

          Activity

          Hide
          ramkrishna.s.vasudevan added a comment -

          This got committed to 0.90.6.
          Sorry for not resolving it at that time.

          Show
          ramkrishna.s.vasudevan added a comment - This got committed to 0.90.6. Sorry for not resolving it at that time.
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Lars
          I understand your point. But as this JIRA is now committed in 0.90 so this addendum is needed.
          I think we can ensure such things doesn't happen for other JIRAs.
          Good on you Lars.

          Show
          ramkrishna.s.vasudevan added a comment - @Lars I understand your point. But as this JIRA is now committed in 0.90 so this addendum is needed. I think we can ensure such things doesn't happen for other JIRAs. Good on you Lars.
          Hide
          Lars Hofhansl added a comment -

          @Ram: Sure. I now feel the whole logic is too complicated, but we'll not fix that any time soon. So +1 on addendum.

          Show
          Lars Hofhansl added a comment - @Ram: Sure. I now feel the whole logic is too complicated, but we'll not fix that any time soon. So +1 on addendum.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512291/HBASE-5153_addendum_0.90_1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/864//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512291/HBASE-5153_addendum_0.90_1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/864//console This message is automatically generated.
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Lars
          Addressing your comments.
          Based on your feedback will commit the patch if it is ok.

          Show
          ramkrishna.s.vasudevan added a comment - @Lars Addressing your comments. Based on your feedback will commit the patch if it is ok.
          Hide
          Lars Hofhansl added a comment -

          That is just guard against closed = true in HConnectionImplementation.abort().
          abort probably not set closed to true, but then a bunch of other places would need to be changed too.

          Show
          Lars Hofhansl added a comment - That is just guard against closed = true in HConnectionImplementation.abort(). abort probably not set closed to true, but then a bunch of other places would need to be changed too.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512248/5153-trunk-minimal.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAdmin
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/861//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/861//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/861//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512248/5153-trunk-minimal.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/861//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/861//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/861//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          For resetConnection() in the new patch, I see:

          +      this.closed = false;
          

          In HConnectionImplementation.close(), we have:

                if (master != null) {
                  if (stopProxy) {
                    HBaseRPC.stopProxy(master);
                  }
                  master = null;
          

          This seems to imply that once a connection is closed, we cannot simply change this.closed to false in resetConnection().

          Show
          Ted Yu added a comment - For resetConnection() in the new patch, I see: + this .closed = false ; In HConnectionImplementation.close(), we have: if (master != null ) { if (stopProxy) { HBaseRPC.stopProxy(master); } master = null ; This seems to imply that once a connection is closed, we cannot simply change this.closed to false in resetConnection().
          Hide
          Lars Hofhansl added a comment -

          Noticed one flaw in the 0.90 addendum patch. Should be

          } catch (ZooKeeperConnectionException zkce) {
            if (tries+1 >= this.numRetries) {
              throw zkce;
            }
          }
          

          (note the +1 in there)

          Show
          Lars Hofhansl added a comment - Noticed one flaw in the 0.90 addendum patch. Should be } catch (ZooKeeperConnectionException zkce) { if (tries+1 >= this .numRetries) { throw zkce; } } (note the +1 in there)
          Hide
          Lars Hofhansl added a comment -

          Here's a minimal patch for trunk.

          If the HConnection is not usable for any reason, the using application can attempt to reset it.
          I kept CloseConnectionException and the idea that ZooKeeperWatcher can unregister a listener.

          If ZK is really not available that is probably an event that the application has to deal with anyway.

          Long term HConnection should not hold persistent connections to ZK anyway, but only establish a connection when needed.

          Show
          Lars Hofhansl added a comment - Here's a minimal patch for trunk. If the HConnection is not usable for any reason, the using application can attempt to reset it. I kept CloseConnectionException and the idea that ZooKeeperWatcher can unregister a listener. If ZK is really not available that is probably an event that the application has to deal with anyway. Long term HConnection should not hold persistent connections to ZK anyway, but only establish a connection when needed.
          Hide
          Ted Yu added a comment -

          Dropping Fix versions as Lars suggested.

          Show
          Ted Yu added a comment - Dropping Fix versions as Lars suggested.
          Hide
          Lars Hofhansl added a comment -

          I take this back... The 0.90 addendum is good. If abort is called from the reset method it'll return right away, and in ZooKeeperNodeTracker.start fall through to return false.

          This entire thing is too complicated. Let's hold off on the 0.92 and trunk patches, but leave it in 0.90 (although I am not a big fan of diverging code bases either).

          Show
          Lars Hofhansl added a comment - I take this back... The 0.90 addendum is good. If abort is called from the reset method it'll return right away, and in ZooKeeperNodeTracker.start fall through to return false. This entire thing is too complicated. Let's hold off on the 0.92 and trunk patches, but leave it in 0.90 (although I am not a big fan of diverging code bases either).
          Hide
          Lars Hofhansl added a comment -

          Actually then the addendum is not right either. If ZooKeeperNodeTracker.start gets an exception it'll call abort (and hence not return false)

          Show
          Lars Hofhansl added a comment - Actually then the addendum is not right either. If ZooKeeperNodeTracker.start gets an exception it'll call abort (and hence not return false)
          Hide
          Lars Hofhansl added a comment -

          @Jieshan... I see your point. Need to think about that.

          Show
          Lars Hofhansl added a comment - @Jieshan... I see your point. Need to think about that.
          Hide
          Lars Hofhansl added a comment -

          +1 on addendum. Technically we should change ZooKeeperNodeTracker.start back to be void, but that is not necessary.
          I'll update the trunk patch soon.

          Show
          Lars Hofhansl added a comment - +1 on addendum. Technically we should change ZooKeeperNodeTracker.start back to be void, but that is not necessary. I'll update the trunk patch soon.
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Ted
          Thanks for the review.
          @Lars
          Can you review the 0.90 patch and provide your comments.
          If it is ok i can integrate to 0.90 and take another RC.

          Show
          ramkrishna.s.vasudevan added a comment - @Ted Thanks for the review. @Lars Can you review the 0.90 patch and provide your comments. If it is ok i can integrate to 0.90 and take another RC.
          Hide
          Ted Yu added a comment -

          +1 on addendum for 0.90

          Show
          Ted Yu added a comment - +1 on addendum for 0.90
          Hide
          Jieshan Bean added a comment -

          This is the addendum for 90.

          Show
          Jieshan Bean added a comment - This is the addendum for 90.
          Hide
          Jieshan Bean added a comment -

          @Lars:
          One more doubt basing on the previous comment, regading on the below code:

          +          try {
          +            setupZookeeperTrackers();
          +            break;
          +          } catch (ZooKeeperConnectionException zkce) {
          +            if (tries >= this.numRetries) {
          +              throw zkce;
          +            }
          +          }
          

          if ZookeeperNodeTracker#start get an exception, then catches and calls abort(Doesn't throw it). The above retries will be break under this situation. So this retries takes no effects.Correct me if am wrong.

          Show
          Jieshan Bean added a comment - @Lars: One more doubt basing on the previous comment, regading on the below code: + try { + setupZookeeperTrackers(); + break; + } catch (ZooKeeperConnectionException zkce) { + if (tries >= this.numRetries) { + throw zkce; + } + } if ZookeeperNodeTracker#start get an exception, then catches and calls abort(Doesn't throw it). The above retries will be break under this situation. So this retries takes no effects.Correct me if am wrong.
          Hide
          Jieshan Bean added a comment -

          @Lars:
          During the retry, if we get any exceptions, Zookeeper and the Trackers also need to close.

          +          try {
          +            setupZookeeperTrackers();
          +            break;
          +          } catch (ZooKeeperConnectionException zkce) {
          +            if (tries >= this.numRetries) {
          +              throw zkce;
          +            }
          +          }
          
          Show
          Jieshan Bean added a comment - @Lars: During the retry, if we get any exceptions, Zookeeper and the Trackers also need to close. + try { + setupZookeeperTrackers(); + break; + } catch (ZooKeeperConnectionException zkce) { + if (tries >= this.numRetries) { + throw zkce; + } + }
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512077/5153-trunk-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 162 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/857//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/857//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/857//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512077/5153-trunk-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 162 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/857//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/857//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/857//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          @Jieshan: Hmm... I do see the endless loop in the debugger. It happens from HBaseAdmin.checkHBaseAvailable when HBase is actually down. We get into resetZooKeeperTrackersWithRetries, which calls setupZookeeperTrackers, which causes abort to be called, which calls resetZooKeeperTrackersWithRetries.
          I am not sure getZooKeeperWatcher() would throw if ZK is not available.

          @Ted: calling this.zooKeeper.close() (if it is not null) first seems prudent.

          @Jieshan: Good catch, yes it should be volatile.

          Show
          Lars Hofhansl added a comment - @Jieshan: Hmm... I do see the endless loop in the debugger. It happens from HBaseAdmin.checkHBaseAvailable when HBase is actually down. We get into resetZooKeeperTrackersWithRetries, which calls setupZookeeperTrackers, which causes abort to be called, which calls resetZooKeeperTrackersWithRetries. I am not sure getZooKeeperWatcher() would throw if ZK is not available. @Ted: calling this.zooKeeper.close() (if it is not null) first seems prudent. @Jieshan: Good catch, yes it should be volatile.
          Hide
          Jieshan Bean added a comment -

          @Lars:
          Adding a "isResettingZKTrackers" sounds good to me. One doubt: Is it necessary to add the keyword of "volatile"?

          Show
          Jieshan Bean added a comment - @Lars: Adding a "isResettingZKTrackers" sounds good to me. One doubt: Is it necessary to add the keyword of "volatile"?
          Hide
          Ted Yu added a comment -
          +        if (isResettingZKTrackers) {
          +          return;
          +        }
          

          I was thinking about something similar to the above.

          In resetZooKeeperTrackersWithRetries(), shall we call zooKeeper.close() before resetting zooKeeper?

                  this.zooKeeper.close();
                  this.zooKeeper = null;
          

          Thanks for the help.

          I think we need an addendum for 0.90

          Show
          Ted Yu added a comment - + if (isResettingZKTrackers) { + return ; + } I was thinking about something similar to the above. In resetZooKeeperTrackersWithRetries(), shall we call zooKeeper.close() before resetting zooKeeper? this .zooKeeper.close(); this .zooKeeper = null ; Thanks for the help. I think we need an addendum for 0.90
          Hide
          Jieshan Bean added a comment -

          Are you guys saying we do not need this in 0.92+?

          We need this, but the patch for 0.92+ should take notice of this

          Show
          Jieshan Bean added a comment - Are you guys saying we do not need this in 0.92+? We need this, but the patch for 0.92+ should take notice of this
          Hide
          Jieshan Bean added a comment -

          "The endless loop happens when ZK is actually down."
          If ZK is actually down, the below code will throw a Exception:
          this.zooKeeper = getZooKeeperWatcher();
          Then catched by the below code:

            try {
                    LOG.info("This client just lost it's session with ZooKeeper, trying" +
                        " to reconnect.");
                    resetZooKeeperTrackersWithRetries();
                    LOG.info("Reconnected successfully. This disconnect could have been" +
                        " caused by a network partition or a long-running GC pause," +
                        " either way it's recommended that you verify your environment.");
                    return;
                  } catch (ZooKeeperConnectionException e) {
                    LOG.error("Could not reconnect to ZooKeeper after session" +
                        " expiration, aborting");
                    t = e;
                  }
                if (t != null) LOG.fatal(msg, t);
                else LOG.fatal(msg);
                HConnectionManager.deleteStaleConnection(this);
          

          It should not be a endless loop. Does that make sense?

          Show
          Jieshan Bean added a comment - "The endless loop happens when ZK is actually down." If ZK is actually down, the below code will throw a Exception: this.zooKeeper = getZooKeeperWatcher(); Then catched by the below code: try { LOG.info("This client just lost it's session with ZooKeeper, trying" + " to reconnect."); resetZooKeeperTrackersWithRetries(); LOG.info("Reconnected successfully. This disconnect could have been" + " caused by a network partition or a long-running GC pause," + " either way it's recommended that you verify your environment."); return; } catch (ZooKeeperConnectionException e) { LOG.error("Could not reconnect to ZooKeeper after session" + " expiration, aborting"); t = e; } if (t != null) LOG.fatal(msg, t); else LOG.fatal(msg); HConnectionManager.deleteStaleConnection(this); It should not be a endless loop. Does that make sense?
          Hide
          Jieshan Bean added a comment -

          Thanks Lars. It's nice of you

          Show
          Jieshan Bean added a comment - Thanks Lars. It's nice of you
          Hide
          Lars Hofhansl added a comment -

          As discussed with Ted. Trunk and 92 already including a retry logic in RecoverableZooKeeper. So that makes the retry logic in resetZooKeeperTrackersWithRetries less important.

          Are you guys saying we do not need this in 0.92+?

          Show
          Lars Hofhansl added a comment - As discussed with Ted. Trunk and 92 already including a retry logic in RecoverableZooKeeper. So that makes the retry logic in resetZooKeeperTrackersWithRetries less important. Are you guys saying we do not need this in 0.92+?
          Hide
          Lars Hofhansl added a comment -

          @Jieshan: The endless loop happens when ZK is actually down.

          Show
          Lars Hofhansl added a comment - @Jieshan: The endless loop happens when ZK is actually down.
          Hide
          Lars Hofhansl added a comment -

          Please let me know what you think of 5153-trunk-v2.txt... Thanks

          Show
          Lars Hofhansl added a comment - Please let me know what you think of 5153-trunk-v2.txt... Thanks
          Hide
          Lars Hofhansl added a comment -

          There was also some weird stuff in HBaseAdmin.checkHBaseAvailable. It did set the client retry to 1, but left the the retry count on in RecoverableZookeeper, which leads to long, unnecessary waits.

          Show
          Lars Hofhansl added a comment - There was also some weird stuff in HBaseAdmin.checkHBaseAvailable. It did set the client retry to 1, but left the the retry count on in RecoverableZookeeper, which leads to long, unnecessary waits.
          Hide
          Lars Hofhansl added a comment -

          I have a new patch... Testing it now.
          It is mostly what you have Jieshan, but it does not need all the changes to the ZookeeperNodeTracker and subclasses.
          I'll attach it soon... Then please let me know what you think.

          Show
          Lars Hofhansl added a comment - I have a new patch... Testing it now. It is mostly what you have Jieshan, but it does not need all the changes to the ZookeeperNodeTracker and subclasses. I'll attach it soon... Then please let me know what you think.
          Hide
          Jieshan Bean added a comment -

          Thanks, Ted...I will take a look at that stack trace.

          If a ZooKeeperConnectionException thrown by the below code:

                 try {
                    if (setupZookeeperTrackers(isLastTime)) {
                      break;
                    }
                  } catch (ZooKeeperConnectionException zkce) {
                    if (isLastTime) {
                      throw zkce;
                    }
                  }
          

          If will be catched in abort method, then calling LOG.fatal(msg, t);

          No problem here. Don't know whether I get you correctly.

          Show
          Jieshan Bean added a comment - Thanks, Ted...I will take a look at that stack trace. If a ZooKeeperConnectionException thrown by the below code: try { if (setupZookeeperTrackers(isLastTime)) { break; } } catch (ZooKeeperConnectionException zkce) { if (isLastTime) { throw zkce; } } If will be catched in abort method, then calling LOG.fatal(msg, t); No problem here. Don't know whether I get you correctly .
          Hide
          Ted Yu added a comment -

          See the stack trace I pasted here:
          https://issues.apache.org/jira/browse/HBASE-5153?focusedCommentId=13187774&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13187774

          If this exception happened for long time, Zookeeper must has some problem.

          We should be prepared when the above indeed happens. I am sure this scenario is possible.

          See also this part of the code:

          +        } catch (ZooKeeperConnectionException zkce) {
          +          if (isLastTime) {
          +            throw zkce;
          +          }
          +        }
          
          Show
          Ted Yu added a comment - See the stack trace I pasted here: https://issues.apache.org/jira/browse/HBASE-5153?focusedCommentId=13187774&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13187774 If this exception happened for long time, Zookeeper must has some problem. We should be prepared when the above indeed happens. I am sure this scenario is possible. See also this part of the code: + } catch (ZooKeeperConnectionException zkce) { + if (isLastTime) { + throw zkce; + } + }
          Hide
          Jieshan Bean added a comment -

          As discussed with Ted. Trunk and 92 already including a retry logic in RecoverableZooKeeper. So that makes the retry logic in resetZooKeeperTrackersWithRetries less important.

          Show
          Jieshan Bean added a comment - As discussed with Ted. Trunk and 92 already including a retry logic in RecoverableZooKeeper. So that makes the retry logic in resetZooKeeperTrackersWithRetries less important.
          Hide
          Jieshan Bean added a comment -

          It should not leading to an endless loop. Unless, each retry will get a ZookeeperLossException. If this exception happened for long time, Zookeeper must has some problem. so when create a new Zookeeper instance, it already thrown a Exception. So it won't be an endless loop:

          if ((t instanceof KeeperException.SessionExpiredException)
                    || (t instanceof KeeperException.ConnectionLossException)) {
                  try {
                    LOG.info("This client just lost it's session with ZooKeeper, trying" +
                        " to reconnect.");
                    resetZooKeeperTrackersWithRetries();
                    LOG.info("Reconnected successfully. This disconnect could have been" +
                        " caused by a network partition or a long-running GC pause," +
                        " either way it's recommended that you verify your environment.");
                    return;
                  } catch (ZooKeeperConnectionException e) {
                    LOG.error("Could not reconnect to ZooKeeper after session" +
                        " expiration, aborting");
                    t = e;
                  }
                }
                if (t != null) LOG.fatal(msg, t);
                else LOG.fatal(msg);
                HConnectionManager.deleteStaleConnection(this);
          
          Show
          Jieshan Bean added a comment - It should not leading to an endless loop. Unless, each retry will get a ZookeeperLossException. If this exception happened for long time, Zookeeper must has some problem. so when create a new Zookeeper instance, it already thrown a Exception. So it won't be an endless loop: if ((t instanceof KeeperException.SessionExpiredException) || (t instanceof KeeperException.ConnectionLossException)) { try { LOG.info("This client just lost it's session with ZooKeeper, trying" + " to reconnect."); resetZooKeeperTrackersWithRetries(); LOG.info("Reconnected successfully. This disconnect could have been" + " caused by a network partition or a long-running GC pause," + " either way it's recommended that you verify your environment."); return; } catch (ZooKeeperConnectionException e) { LOG.error("Could not reconnect to ZooKeeper after session" + " expiration, aborting"); t = e; } } if (t != null) LOG.fatal(msg, t); else LOG.fatal(msg); HConnectionManager.deleteStaleConnection(this);
          Hide
          Lars Hofhansl added a comment -

          Sure... There's a bit more to this too. resetZooKeeperTrackersWithRetries on its last try calls setupZookeeperTrackers with allow aborts, which will call resetZooKeeperTrackersWithRetries again. Leading to an endless loop. Need to think about how to refactor this.

          Show
          Lars Hofhansl added a comment - Sure... There's a bit more to this too. resetZooKeeperTrackersWithRetries on its last try calls setupZookeeperTrackers with allow aborts, which will call resetZooKeeperTrackersWithRetries again. Leading to an endless loop. Need to think about how to refactor this.
          Hide
          Ted Yu added a comment -

          Thanks for tracking down the issue, Lars.
          If you can upload the latest 5153-trunk.txt to reviewboard first followed by your new patch, that would help us know your changes easily.

          Show
          Ted Yu added a comment - Thanks for tracking down the issue, Lars. If you can upload the latest 5153-trunk.txt to reviewboard first followed by your new patch, that would help us know your changes easily.
          Hide
          Lars Hofhansl added a comment -

          So here's the problem. This is hanging while validating that HBase is not running via HBaseAdmin.checkHBaseAvailable, which just attempts to create a new HBaseAdmin after it sets hbase.client.retries.number to 1. However HConnectionImpl caches hbase.client.retries.number in numRetries, and hence if ZK is not running resetZooKeeperTrackersWithRetries will retry for a while.
          The simplest fix would be for resetZooKeeperTrackersWithRetries to ignore he cached setting and to retrieve the value again from the setting. While I am at it, I'll also add another option to a different number of retries here.

          Show
          Lars Hofhansl added a comment - So here's the problem. This is hanging while validating that HBase is not running via HBaseAdmin.checkHBaseAvailable, which just attempts to create a new HBaseAdmin after it sets hbase.client.retries.number to 1. However HConnectionImpl caches hbase.client.retries.number in numRetries, and hence if ZK is not running resetZooKeeperTrackersWithRetries will retry for a while. The simplest fix would be for resetZooKeeperTrackersWithRetries to ignore he cached setting and to retrieve the value again from the setting. While I am at it, I'll also add another option to a different number of retries here.
          Hide
          Lars Hofhansl added a comment -

          So is this change in 0.90 now? I'm confused. Should revert it from there too, I guess.
          I will see what's up with TestMergeTool in trunk now.

          Show
          Lars Hofhansl added a comment - So is this change in 0.90 now? I'm confused. Should revert it from there too, I guess. I will see what's up with TestMergeTool in trunk now.
          Hide
          Ted Yu added a comment -

          There were two failed tests:
          https://builds.apache.org/job/HBase-0.92-security/81/

          If you can resolve the hanging TestMergeTool, that would be great.

          I am on-call this week, FYI

          Show
          Ted Yu added a comment - There were two failed tests: https://builds.apache.org/job/HBase-0.92-security/81/ If you can resolve the hanging TestMergeTool, that would be great. I am on-call this week, FYI
          Hide
          Lars Hofhansl added a comment -

          Also it occurred to me that another nice change would be to be able specify the retry count for resetZooKeeperTrackersWithRetries different from the other operations.
          The thinking is this:
          While the ZK is not reachable the HConnection (and any other HConnection) is essentially not usable. In some settings it might be good to have the connection just sit there, and retry until the connection is bad. Maybe for another jira.

          Where are we with this generally?
          Is it just TestMergeTool hanging? If so I'll have a look at it today.

          Show
          Lars Hofhansl added a comment - Also it occurred to me that another nice change would be to be able specify the retry count for resetZooKeeperTrackersWithRetries different from the other operations. The thinking is this: While the ZK is not reachable the HConnection (and any other HConnection) is essentially not usable. In some settings it might be good to have the connection just sit there, and retry until the connection is bad. Maybe for another jira. Where are we with this generally? Is it just TestMergeTool hanging? If so I'll have a look at it today.
          Hide
          Krystian Nowak added a comment -
          Show
          Krystian Nowak added a comment - FYI: HBASE-5153-V6-90-minorchange.patch also fixes HBASE-5289 (linked)
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2639 (See https://builds.apache.org/job/HBase-TRUNK/2639/)
          HBASE-5153 revert due to failed Jenkins builds

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2639 (See https://builds.apache.org/job/HBase-TRUNK/2639/ ) HBASE-5153 revert due to failed Jenkins builds tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #81 (See https://builds.apache.org/job/HBase-TRUNK-security/81/)
          HBASE-5153 revert due to failed Jenkins builds

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #81 (See https://builds.apache.org/job/HBase-TRUNK-security/81/ ) HBASE-5153 revert due to failed Jenkins builds tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #83 (See https://builds.apache.org/job/HBase-0.92-security/83/)
          HBASE-5153 revert due to failed Jenkins builds

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #83 (See https://builds.apache.org/job/HBase-0.92-security/83/ ) HBASE-5153 revert due to failed Jenkins builds tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #248 (See https://builds.apache.org/job/HBase-0.92/248/)
          HBASE-5153 revert due to failed Jenkins builds

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #248 (See https://builds.apache.org/job/HBase-0.92/248/ ) HBASE-5153 revert due to failed Jenkins builds tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Ted Yu added a comment -

          TestMergeTool hung in several builds.
          That was why I ran it on MacBook.

          Show
          Ted Yu added a comment - TestMergeTool hung in several builds. That was why I ran it on MacBook.
          Hide
          Lars Hofhansl added a comment -

          Was that repeatable? Might be a problem with the test (relying on the fact that an HConnection would just give up after one try).

          Show
          Lars Hofhansl added a comment - Was that repeatable? Might be a problem with the test (relying on the fact that an HConnection would just give up after one try).
          Hide
          Ted Yu added a comment -

          Reverted from 0.92 and TRUNK due to failed Jenkins builds.

          Show
          Ted Yu added a comment - Reverted from 0.92 and TRUNK due to failed Jenkins builds.
          Hide
          Ted Yu added a comment -

          When I ran TestMergeTool on TRUNK, I got:

          "main" prio=5 tid=102801000 nid=0x100601000 waiting on condition [1005fb000]
             java.lang.Thread.State: TIMED_WAITING (sleeping)
          	at java.lang.Thread.sleep(Native Method)
          	at java.lang.Thread.sleep(Thread.java:302)
          	at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:328)
          	at org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:55)
          	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:171)
          	at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:80)
          	- locked <784a83930> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <7854b9898> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <7854b98e0> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <7854b9928> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78f991ad0> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790b9f540> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790b9f560> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790b9f580> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790b9f5a0> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78f98ae20> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78f98ae40> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78f98ae60> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78e8b6ac8> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78e8b6ae8> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <78e8b6b08> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e640> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e660> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e600> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e620> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e5c0> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e5e0> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e580> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e5a0> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e540> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e560> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711)
          	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93)
          	- locked <790d0e500> (a org.apache.hadoop.hbase.MasterAddressTracker)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590)
          	- locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:577)
          	at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:184)
          	- locked <790c46258> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
          	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:98)
          	at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1570)
          	at org.apache.hadoop.hbase.util.Merge.run(Merge.java:94)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
          	at org.apache.hadoop.hbase.util.TestMergeTool.mergeAndVerify(TestMergeTool.java:190)
          	at org.apache.hadoop.hbase.util.TestMergeTool.testMergeTool(TestMergeTool.java:268)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at junit.framework.TestCase.runTest(TestCase.java:168)
          	at junit.framework.TestCase.runBare(TestCase.java:134)
          	at junit.framework.TestResult$1.protect(TestResult.java:110)
          	at junit.framework.TestResult.runProtected(TestResult.java:128)
          	at junit.framework.TestResult.run(TestResult.java:113)
          	at junit.framework.TestCase.run(TestCase.java:124)
          	at junit.framework.TestSuite.runTest(TestSuite.java:243)
          	at junit.framework.TestSuite.run(TestSuite.java:238)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
          	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
          	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
          	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
          	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
          
          Show
          Ted Yu added a comment - When I ran TestMergeTool on TRUNK, I got: "main" prio=5 tid=102801000 nid=0x100601000 waiting on condition [1005fb000] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at java.lang. Thread .sleep( Thread .java:302) at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:328) at org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:55) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:171) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:80) - locked <784a83930> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <7854b9898> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <7854b98e0> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <7854b9928> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78f991ad0> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790b9f540> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790b9f560> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790b9f580> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790b9f5a0> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78f98ae20> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78f98ae40> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78f98ae60> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78e8b6ac8> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78e8b6ae8> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <78e8b6b08> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e640> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e660> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e600> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e620> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e5c0> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e5e0> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e580> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e5a0> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e540> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e560> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackersWithRetries(HConnectionManager.java:625) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1711) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:93) - locked <790d0e500> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:590) - locked <790b70180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:577) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:184) - locked <790c46258> (a org.apache.hadoop.hbase.client.HConnectionManager$1) at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:98) at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1570) at org.apache.hadoop.hbase.util.Merge.run(Merge.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.TestMergeTool.mergeAndVerify(TestMergeTool.java:190) at org.apache.hadoop.hbase.util.TestMergeTool.testMergeTool(TestMergeTool.java:268) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #81 (See https://builds.apache.org/job/HBase-0.92-security/81/)
          HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan)

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #81 (See https://builds.apache.org/job/HBase-0.92-security/81/ ) HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #246 (See https://builds.apache.org/job/HBase-0.92/246/)
          HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan)

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #246 (See https://builds.apache.org/job/HBase-0.92/246/ ) HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #79 (See https://builds.apache.org/job/HBase-TRUNK-security/79/)
          HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan)

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #79 (See https://builds.apache.org/job/HBase-TRUNK-security/79/ ) HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan) tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2635 (See https://builds.apache.org/job/HBase-TRUNK/2635/)
          HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan)

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2635 (See https://builds.apache.org/job/HBase-TRUNK/2635/ ) HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan) tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
          Hide
          ramkrishna.s.vasudevan added a comment -

          Thanks a lot Ted. Nice of you .

          Show
          ramkrishna.s.vasudevan added a comment - Thanks a lot Ted. Nice of you .
          Hide
          Jieshan Bean added a comment -

          Thank you, Ted.

          Show
          Jieshan Bean added a comment - Thank you, Ted.
          Hide
          Ted Yu added a comment -

          Integrated to 0.92 branch.

          Show
          Ted Yu added a comment - Integrated to 0.92 branch.
          Hide
          Ted Yu added a comment -

          Patch for 0.92 branch

          Show
          Ted Yu added a comment - Patch for 0.92 branch
          Hide
          Ted Yu added a comment -

          Integrated to 0.90 and TRUNK.

          Waiting for 0.92 security build to pass before integrating to 0.92

          Thanks for the patch Jieshan.

          Thanks for the review Lars.

          Show
          Ted Yu added a comment - Integrated to 0.90 and TRUNK. Waiting for 0.92 security build to pass before integrating to 0.92 Thanks for the patch Jieshan. Thanks for the review Lars.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510779/5153-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -144 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 83 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplication
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/783//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/783//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/783//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510779/5153-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -144 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 83 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/783//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/783//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/783//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          +1 on trunk patch. 0.92 is similar?

          Show
          Lars Hofhansl added a comment - +1 on trunk patch. 0.92 is similar?
          Hide
          Ted Yu added a comment -

          Patch for TRUNK.

          Show
          Ted Yu added a comment - Patch for TRUNK.
          Hide
          Lars Hofhansl added a comment -

          +1 on latest patch

          Show
          Lars Hofhansl added a comment - +1 on latest patch
          Hide
          Ted Yu added a comment -

          @Jieshan:
          Please find a machine which has access to internet to run the test suite.
          maven needs to download artifacts.

          I ran the 5 tests on MacBook and they passed.

            839  mt -Dtest=TestClockSkewDetection
            840  mt -Dtest=TestScanner
            841  mt -Dtest=TestCatalogTrackerOnCluster
            842  mt -Dtest=TestCatalogTracker
          

          +1 on latest patch.

          Show
          Ted Yu added a comment - @Jieshan: Please find a machine which has access to internet to run the test suite. maven needs to download artifacts. I ran the 5 tests on MacBook and they passed. 839 mt -Dtest=TestClockSkewDetection 840 mt -Dtest=TestScanner 841 mt -Dtest=TestCatalogTrackerOnCluster 842 mt -Dtest=TestCatalogTracker +1 on latest patch.
          Hide
          Jieshan Bean added a comment -

          Ran the tests again, got the same results: 5 tests failed due to the hostName problem. Please find the results from the attachment "TestResults-hbase5153.out".

          Show
          Jieshan Bean added a comment - Ran the tests again, got the same results: 5 tests failed due to the hostName problem. Please find the results from the attachment "TestResults-hbase5153.out".
          Hide
          Jieshan Bean added a comment -

          I will upload the new patch after the tests finish(Including this minor change), Thanks, Ted.

          Show
          Jieshan Bean added a comment - I will upload the new patch after the tests finish(Including this minor change), Thanks, Ted.
          Hide
          Jieshan Bean added a comment -

          The test is still running. Before I get the results, I want your comments.
          Thank you.

          Show
          Jieshan Bean added a comment - The test is still running. Before I get the results, I want your comments . Thank you.
          Hide
          Ted Yu added a comment -

          Minor comment:
          In setupZookeeperTrackers(), after calling stop(), the masterAddressTracker / rootRegionTracker should be set to null.

          Please run patch v6 over test suite.

          Thanks

          Show
          Ted Yu added a comment - Minor comment: In setupZookeeperTrackers(), after calling stop(), the masterAddressTracker / rootRegionTracker should be set to null. Please run patch v6 over test suite. Thanks
          Hide
          Jieshan Bean added a comment -

          Changes:
          1. Add a method "unregisterListener" in ZookeeperWatcher. This will be called in ZookeeperNodeTracker's stop method.
          2. Stop the trackers if start failed during the retries.

          Thank you , Ted.

          @Lars, I will make the patches for trunk and 92, patch for 90 seems more urgent..Thank you.

          Show
          Jieshan Bean added a comment - Changes: 1. Add a method "unregisterListener" in ZookeeperWatcher. This will be called in ZookeeperNodeTracker's stop method. 2. Stop the trackers if start failed during the retries. Thank you , Ted. @Lars, I will make the patches for trunk and 92, patch for 90 seems more urgent ..Thank you.
          Hide
          Lars Hofhansl added a comment -

          Oh, and should probably change the title of this jira.

          Show
          Lars Hofhansl added a comment - Oh, and should probably change the title of this jira.
          Hide
          Lars Hofhansl added a comment - - edited

          Thanks Jieshan. This patch is great.
          +1 after Ted's comments are fixed.

          Are you planning to make a 0.92 and trunk patch as well?
          (After Stacks changes in trunk that might be not completely trivial).

          Show
          Lars Hofhansl added a comment - - edited Thanks Jieshan. This patch is great. +1 after Ted's comments are fixed. Are you planning to make a 0.92 and trunk patch as well? (After Stacks changes in trunk that might be not completely trivial).
          Hide
          Ted Yu added a comment -
          +   * Close the original connection and create a new one.
          +   * @throws ZooKeeperConnectionException  if unable to connect to zookeeper
          +   */
          +  public void resetConnection() throws ZooKeeperConnectionException;
          

          The first line should read 'Closes the original connection and creates a new one.'
          Since resetZooKeeperTrackersWithRetries() is called underneath, I think naming this new method resetZooKeeperTrackersWithRetries() seems better.

          +/**
          + * Thrown when HConnection has been closed.
          + */
          +public class InvalidConnectionException extends IOException {
          

          Going over the patch, the new exception is indeed used to indicate closed connection. I think naming it ClosedConnectionException is better.

          Show
          Ted Yu added a comment - + * Close the original connection and create a new one. + * @ throws ZooKeeperConnectionException if unable to connect to zookeeper + */ + public void resetConnection() throws ZooKeeperConnectionException; The first line should read 'Closes the original connection and creates a new one.' Since resetZooKeeperTrackersWithRetries() is called underneath, I think naming this new method resetZooKeeperTrackersWithRetries() seems better. +/** + * Thrown when HConnection has been closed. + */ + public class InvalidConnectionException extends IOException { Going over the patch, the new exception is indeed used to indicate closed connection. I think naming it ClosedConnectionException is better.
          Hide
          Ted Yu added a comment -

          For patch v5, setupZookeeperTrackers():

          -      this.rootRegionTracker.start();
          +      return this.rootRegionTracker.start(allowAbort)
          +          && this.masterAddressTracker.start(allowAbort);
          

          If rootRegionTracker.start() returns false, masterAddressTracker.start() would be skipped.
          Since setupZookeeperTrackers() is called by HConnectionImplementation ctor, I suggest making setupZookeeperTrackers() reentrant (checking masterAddressTracker and rootRegionTracker against null) and calling it in HConnectionImplementation ctor.

          Show
          Ted Yu added a comment - For patch v5, setupZookeeperTrackers(): - this .rootRegionTracker.start(); + return this .rootRegionTracker.start(allowAbort) + && this .masterAddressTracker.start(allowAbort); If rootRegionTracker.start() returns false, masterAddressTracker.start() would be skipped. Since setupZookeeperTrackers() is called by HConnectionImplementation ctor, I suggest making setupZookeeperTrackers() reentrant (checking masterAddressTracker and rootRegionTracker against null) and calling it in HConnectionImplementation ctor.
          Hide
          Jieshan Bean added a comment -

          This is the new patch for 90 according to Lars' suggestion. Tell me if anything still need to change.

          Thank you Ted, Lars & Ram.

          Show
          Jieshan Bean added a comment - This is the new patch for 90 according to Lars' suggestion. Tell me if anything still need to change. Thank you Ted, Lars & Ram.
          Hide
          ramkrishna.s.vasudevan added a comment -

          No problem Lars. Its good to get the best suggestions and ideas.

          Show
          ramkrishna.s.vasudevan added a comment - No problem Lars. Its good to get the best suggestions and ideas.
          Hide
          Lars Hofhansl added a comment -

          Thanks Ram (and Jieshan and Ted).
          In my previous comments I did not mean to be abrasive. Thanks for finding this problem and working on a patch for it (I completely missed this when I worked on HBASE-4805).

          Show
          Lars Hofhansl added a comment - Thanks Ram (and Jieshan and Ted). In my previous comments I did not mean to be abrasive. Thanks for finding this problem and working on a patch for it (I completely missed this when I worked on HBASE-4805 ).
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Lars
          Ok Lars. Tomorrow may be we can upload a new patch with a retry logic in resetZooKeeperTrackers. I will discuss with Jieshan tomorrow. Thanks Lars.

          Show
          ramkrishna.s.vasudevan added a comment - @Lars Ok Lars. Tomorrow may be we can upload a new patch with a retry logic in resetZooKeeperTrackers. I will discuss with Jieshan tomorrow. Thanks Lars.
          Hide
          Lars Hofhansl added a comment -

          @Ted: This could go into 0.90. Downside then is that 0.90 would behave quite differently.
          I feel ambivalent about HBASE-5084 (seems we're over engineering this, but if you think we need that, then let's fix that first and then revisit this).

          @Jieshan & Ted: Maybe have some retry logic resetZooKeeperTrackers and give HConnection an extra "reset" method? That way app code can redo the reset if needed.

          Show
          Lars Hofhansl added a comment - @Ted: This could go into 0.90. Downside then is that 0.90 would behave quite differently. I feel ambivalent about HBASE-5084 (seems we're over engineering this, but if you think we need that, then let's fix that first and then revisit this). @Jieshan & Ted: Maybe have some retry logic resetZooKeeperTrackers and give HConnection an extra "reset" method? That way app code can redo the reset if needed.
          Hide
          Ted Yu added a comment -

          I think we can solve HBASE-5084 first by not bundling the HConnection and ExecutorService.
          After that, this JIRA's patch for TRUNK can be modified to take effect when HConnection is managed internally.

          Show
          Ted Yu added a comment - I think we can solve HBASE-5084 first by not bundling the HConnection and ExecutorService. After that, this JIRA's patch for TRUNK can be modified to take effect when HConnection is managed internally.
          Hide
          Jieshan Bean added a comment -

          Thanks, Ted.

          Without HBASE-3065, 0.90 doesn't handle the ConnectionLossException correctly. Consider the below case:
          1. Somewhere trigger a HConnection#abort.
          2. Suppose the check of "if (t instanceof KeeperException.SessionExpiredException)" is true. Then called the resetZooKeeperTrackers().
          3. A ConnectionLossException occur during ZookeeperNodeTracker#start. then trigger a new HConnection#abort. At this scenario, the previous abort may print a log of
          "Reconnected successfully. This disconnect could have been caused by a network partition or a long-running GC pause......"
          4. The new abort carry a Throwable with a type which is not KeeperException.SessionExpiredException. so this time abort directly.

          It seems a recursion here.

          Either re-use the old connection by resetZooKeeperTrackers, or re-create the connection, the ZookeeperWatcher will be a new one. So I still think the patch for 0.90 is reasonable.

          Trunk patch will be made big changes.

          So any other good suggestions? Thanks.

          Show
          Jieshan Bean added a comment - Thanks, Ted. Without HBASE-3065 , 0.90 doesn't handle the ConnectionLossException correctly. Consider the below case: 1. Somewhere trigger a HConnection#abort. 2. Suppose the check of "if (t instanceof KeeperException.SessionExpiredException)" is true. Then called the resetZooKeeperTrackers(). 3. A ConnectionLossException occur during ZookeeperNodeTracker#start. then trigger a new HConnection#abort. At this scenario, the previous abort may print a log of "Reconnected successfully. This disconnect could have been caused by a network partition or a long-running GC pause......" 4. The new abort carry a Throwable with a type which is not KeeperException.SessionExpiredException. so this time abort directly. It seems a recursion here. Either re-use the old connection by resetZooKeeperTrackers, or re-create the connection, the ZookeeperWatcher will be a new one. So I still think the patch for 0.90 is reasonable. Trunk patch will be made big changes. So any other good suggestions? Thanks.
          Hide
          Ted Yu added a comment -

          @Lars:
          Hbase-4805 is not in 0.90
          So in spirit, this Jira should be able to go into 0.90
          What do you think ?

          Show
          Ted Yu added a comment - @Lars: Hbase-4805 is not in 0.90 So in spirit, this Jira should be able to go into 0.90 What do you think ?
          Hide
          Jieshan Bean added a comment -

          Maybe retry in resetZooKeeperTrackers is better. We try our best to re-use the original connection. The worst case is even if we retried max times and still fail, then abort, but i think we're responsible for letting the user level know this.

          Show
          Jieshan Bean added a comment - Maybe retry in resetZooKeeperTrackers is better. We try our best to re-use the original connection. The worst case is even if we retried max times and still fail, then abort, but i think we're responsible for letting the user level know this.
          Hide
          Lars Hofhansl added a comment -

          Is the only time when this happens a loss of the ZK connection?
          All that a new HConnectionImplementation is doing is a call to setupZookeeperTrackers.
          HConnectionImplementation.abort already does this. Why would that succeed in the new Connection? Is it because we retry?

          Maybe HConnectionImplementation.resetZooKeeperTrackers should have the retry logic.

          -1 on current patches.

          Show
          Lars Hofhansl added a comment - Is the only time when this happens a loss of the ZK connection? All that a new HConnectionImplementation is doing is a call to setupZookeeperTrackers. HConnectionImplementation.abort already does this. Why would that succeed in the new Connection? Is it because we retry? Maybe HConnectionImplementation.resetZooKeeperTrackers should have the retry logic. -1 on current patches.
          Hide
          Lars Hofhansl added a comment -

          The trunk and 0.92 patch won't work with HBASE-4805.
          I would rather see a way to be able to reuse the connection rather than getting a new one as this defeats the whole idea of HBASE-4805.

          Show
          Lars Hofhansl added a comment - The trunk and 0.92 patch won't work with HBASE-4805 . I would rather see a way to be able to reuse the connection rather than getting a new one as this defeats the whole idea of HBASE-4805 .
          Hide
          Ted Yu added a comment -

          +1 on latest patches.

          Show
          Ted Yu added a comment - +1 on latest patches.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510458/HBASE-5153-trunk-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -147 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 80 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/751//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/751//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/751//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510458/HBASE-5153-trunk-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -147 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 80 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/751//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/751//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/751//console This message is automatically generated.
          Hide
          Jieshan Bean added a comment -

          I updated the patches.

          Show
          Jieshan Bean added a comment - I updated the patches.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510397/5153-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -147 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 80 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplicationPeer
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestImportTsv

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/748//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/748//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/748//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510397/5153-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -147 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 80 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplicationPeer org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/748//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/748//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/748//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          For the trunk patch:

          +   * @param actions
          +   *          The collection of actions.
          +   * @param tableName
          +   *          Name of the hbase table
          

          The explanation should be on the same line as parameter name.

          +   * Check the connection. If it has been closed, we should recreate the
          +   * Connection. See HBASE-5153.
          +   * 
          +   * @param ioe
          +   *          The IOE throwed by the HConnection methods.
          

          Connection on second line above should start with lower case 'c'.
          throwed should be thrown.

          +        throw new RetriesExhaustedException(
          +            "Retry to recreate connection in HTable, but failed.");
          

          Please include the value of tries in the exception message.

          Good work.

          Show
          Ted Yu added a comment - For the trunk patch: + * @param actions + * The collection of actions. + * @param tableName + * Name of the hbase table The explanation should be on the same line as parameter name. + * Check the connection. If it has been closed, we should recreate the + * Connection. See HBASE-5153. + * + * @param ioe + * The IOE throwed by the HConnection methods. Connection on second line above should start with lower case 'c'. throwed should be thrown. + throw new RetriesExhaustedException( + "Retry to recreate connection in HTable, but failed." ); Please include the value of tries in the exception message. Good work.
          Hide
          Ted Yu added a comment -

          Jieshan's patch for TRUNK.

          Show
          Ted Yu added a comment - Jieshan's patch for TRUNK.
          Hide
          Jieshan Bean added a comment -

          @Ted, the patch for TRUNK seems very different, and i still need some time to check it. hope i can provide today

          @Stack, I think ConnectionUtils is reasonable. I can add it. I will update the patch.

          Thank you all.

          Show
          Jieshan Bean added a comment - @Ted, the patch for TRUNK seems very different, and i still need some time to check it. hope i can provide today @Stack, I think ConnectionUtils is reasonable. I can add it . I will update the patch. Thank you all.
          Hide
          stack added a comment -

          Patch looks good. I like your addition of a specific Exception for closed state.

          Does this have to be public Jieshan?

          getRegionServerWithRetries
          

          Same for processBatch and getRegionLocation.

          If public should be in HTableInterface but they seem implementation methods rather than something that should be part of public interface.

          A style nit – i.e. not important but if you are going to redo the patch you miight want to address it – is that you do this in handleConnectionClosedException....

          +    if (ioe instanceof ConnectionClosedException) {
          

          and the whole method is dealing with the case where above is true. I'd suggest that you might do:

          if (!(ioe instanceof ConnectionClosedException)) return;
          

          ... then you save a whole indent and its clear that the method is all about dealing with ConnectionClosedException.

          Is it right including this in HTable?

          getPauseTime
          

          In trunk that is in a new ConnectionUtils class. Maybe you have to do it for 0.90?

          I'm wondering if the class ConnectionClosedException needs to be public also? Its only used in this package, right?

          Show
          stack added a comment - Patch looks good. I like your addition of a specific Exception for closed state. Does this have to be public Jieshan? getRegionServerWithRetries Same for processBatch and getRegionLocation. If public should be in HTableInterface but they seem implementation methods rather than something that should be part of public interface. A style nit – i.e. not important but if you are going to redo the patch you miight want to address it – is that you do this in handleConnectionClosedException.... + if (ioe instanceof ConnectionClosedException) { and the whole method is dealing with the case where above is true. I'd suggest that you might do: if (!(ioe instanceof ConnectionClosedException)) return ; ... then you save a whole indent and its clear that the method is all about dealing with ConnectionClosedException. Is it right including this in HTable? getPauseTime In trunk that is in a new ConnectionUtils class. Maybe you have to do it for 0.90? I'm wondering if the class ConnectionClosedException needs to be public also? Its only used in this package, right?
          Hide
          Ted Yu added a comment -

          @Jieshan:
          Can you prepare a patch for trunk ?

          Show
          Ted Yu added a comment - @Jieshan: Can you prepare a patch for trunk ?
          Hide
          Jieshan Bean added a comment -

          Only do that in flushCommits maybe not enough. I'll go though the code and give a more considerate approach. and also will give a patch for TRUNK.

          Show
          Jieshan Bean added a comment - Only do that in flushCommits maybe not enough. I'll go though the code and give a more considerate approach. and also will give a patch for TRUNK.
          Hide
          Ted Yu added a comment -
          +      if (this.closed)
          +        throw new ConnectionClosedException(toString() + " closed");
          

          Please enclose the throw statement in curly braces.

          Please also prepare patch for TRUNK.

          Show
          Ted Yu added a comment - + if ( this .closed) + throw new ConnectionClosedException(toString() + " closed" ); Please enclose the throw statement in curly braces. Please also prepare patch for TRUNK.
          Hide
          Jieshan Bean added a comment -

          Thanks, Ted. I'll add the unit test code immediately.

          Show
          Jieshan Bean added a comment - Thanks, Ted. I'll add the unit test code immediately.
          Hide
          Ted Yu added a comment -

          Can you add a unit test ?
          Testing this in cluster is also desirable.

          Show
          Ted Yu added a comment - Can you add a unit test ? Testing this in cluster is also desirable.
          Hide
          Jieshan Bean added a comment -

          Thank you, Rama. I update the patch

          Show
          Jieshan Bean added a comment - Thank you, Rama. I update the patch
          Hide
          ramkrishna.s.vasudevan added a comment -

          Your idea is correct.
          Few comments

          Copyright 2010 The Apache Software Foundation
          

          This can be removed.
          Also add javadoc for the new class added.

          Show
          ramkrishna.s.vasudevan added a comment - Your idea is correct. Few comments Copyright 2010 The Apache Software Foundation This can be removed. Also add javadoc for the new class added.
          Hide
          Jieshan Bean added a comment -

          Plz share your comments, thank you.

          Show
          Jieshan Bean added a comment - Plz share your comments, thank you.

            People

            • Assignee:
              Jieshan Bean
              Reporter:
              Jieshan Bean
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development