Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6326

Avoid nested retry loops in HConnectionManager

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None

      Description

      While testing client timeouts when the HBase is not available we found that even with aggressive settings, it takes the client 10 minutes or more to finally receive an exception.
      Part of this is due to nested nested retry loops in locateRegion.

      locateRegion will first try to locate the table in meta (which is retried), then it will try to locate the meta table is root (which is also retried).
      So for each retry of the meta lookup we retry the root lookup as well.

      I have have that avoids locateRegion retrying if it is called from code that already has a retry loop.

      1. 6326-trunk.txt
        4 kB
        Lars Hofhansl
      2. 6326.txt
        4 kB
        Lars Hofhansl

        Activity

        Hide
        hudson Hudson added a comment -

        Integrated in HBase-0.94-security #39 (See https://builds.apache.org/job/HBase-0.94-security/39/)
        HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752)

        Result = SUCCESS
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Show
        hudson Hudson added a comment - Integrated in HBase-0.94-security #39 (See https://builds.apache.org/job/HBase-0.94-security/39/ ) HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Hide
        hudson Hudson added a comment -

        Integrated in HBase-0.94-deploy #4 (See https://builds.apache.org/job/HBase-0.94-deploy/4/)
        HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752)

        Result = SUCCESS
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Show
        hudson Hudson added a comment - Integrated in HBase-0.94-deploy #4 (See https://builds.apache.org/job/HBase-0.94-deploy/4/ ) HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Hide
        hudson Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #5 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/5/)
        HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752)

        Result = FAILURE
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Show
        hudson Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #5 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/5/ ) HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Hide
        hudson Hudson added a comment -

        Integrated in HBase-0.94 #298 (See https://builds.apache.org/job/HBase-0.94/298/)
        HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752)

        Result = FAILURE
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Show
        hudson Hudson added a comment - Integrated in HBase-0.94 #298 (See https://builds.apache.org/job/HBase-0.94/298/ ) HBASE-6326 Avoid nested retry loops in HConnectionManager (Revision 1357752) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        Hide
        lhofhansl Lars Hofhansl added a comment -

        Committed to 0.94 and 0.96.

        Show
        lhofhansl Lars Hofhansl added a comment - Committed to 0.94 and 0.96.
        Hide
        lhofhansl Lars Hofhansl added a comment -

        Patch for trunk.

        Show
        lhofhansl Lars Hofhansl added a comment - Patch for trunk.
        Hide
        lhofhansl Lars Hofhansl added a comment -

        Looking at the code, the problem should exist in 0.96 too.

        Show
        lhofhansl Lars Hofhansl added a comment - Looking at the code, the problem should exist in 0.96 too.
        Hide
        lhofhansl Lars Hofhansl added a comment -

        Yep... Not pretty. Neither is the rest of this.
        I'll check whether this particular issue is fixed in 0.96.
        (The nested retrying might be the same)

        Show
        lhofhansl Lars Hofhansl added a comment - Yep... Not pretty. Neither is the rest of this. I'll check whether this particular issue is fixed in 0.96. (The nested retrying might be the same)
        Hide
        stack stack added a comment -

        +1 Simple but ugly. Good enough for a 0.94.1.

        Show
        stack stack added a comment - +1 Simple but ugly. Good enough for a 0.94.1.
        Hide
        lhofhansl Lars Hofhansl added a comment -

        Patch is against 0.94. 0.96 is quite different.

        Show
        lhofhansl Lars Hofhansl added a comment - Patch is against 0.94. 0.96 is quite different.
        Hide
        lhofhansl Lars Hofhansl added a comment -

        Simple patch avoiding retries in locateRegion is called from locateRegionInMeta, which already has a retry loop.

        Show
        lhofhansl Lars Hofhansl added a comment - Simple patch avoiding retries in locateRegion is called from locateRegionInMeta, which already has a retry loop.
        Hide
        lhofhansl Lars Hofhansl added a comment -

        For us this is pretty critical.

        Show
        lhofhansl Lars Hofhansl added a comment - For us this is pretty critical.

          People

          • Assignee:
            lhofhansl Lars Hofhansl
            Reporter:
            lhofhansl Lars Hofhansl
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development