HBase
  1. HBase
  2. HBASE-6956

Do not return back to HTablePool closed connections

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.90.6
    • Fix Version/s: None
    • Component/s: Client
    • Labels:
      None

      Description

      Sometimes we see a lot of Exception about closed connections:

       org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068 closed
      org.apache.hadoop.hbase.client.ClosedConnectionException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068 closed
      

      After investigation we assumed that it occurs because closed connection returns back into HTablePool.

      For our opinion best solution is check whether the table is closed in method HTablePool.putTable and if true don't add it into the queue and release such HTableInterface.

      But unfortunatly right now there are no access to HTable#closed field through HTableInterface

        Issue Links

          Activity

          Hide
          stack added a comment -

          You have a patch Igor?

          Show
          stack added a comment - You have a patch Igor?
          Hide
          shengjie min added a comment -

          I still have this issue even in 0.92.1. any updates?

          Show
          shengjie min added a comment - I still have this issue even in 0.92.1. any updates?
          Hide
          Ted Yu added a comment -

          For putTable(), I see this javadoc:

            /**
             * This method is not needed anymore, clients should call
             * HTableInterface.close() rather than returning the tables to the pool
             * 
             * @param table
             *          the proxy table user got from pool
             * @deprecated
             */
            public void putTable(HTableInterface table) throws IOException {
          

          @Igor, @Shengjie:
          Can you tell us whether you followed the above suggestion ?

          Thanks

          Show
          Ted Yu added a comment - For putTable(), I see this javadoc: /** * This method is not needed anymore, clients should call * HTableInterface.close() rather than returning the tables to the pool * * @param table * the proxy table user got from pool * @deprecated */ public void putTable(HTableInterface table) throws IOException { @Igor, @Shengjie: Can you tell us whether you followed the above suggestion ? Thanks
          Hide
          shengjie min added a comment -

          In my case I am using HTablePool, my understanding is the following:

          so when the hbase query comes,

          if pool is avaiable:
          I get a PooledHtable instance, once I am done with query, I call HtablePool.PooledHtable.close(), it will return PooledHtable instance to the pool.
          if pool is out:
          I get a normal Htable instance, once I am done with query, I call Htable.close(), it will just release Htable instance.

          I have no clue why I ended up having closed HConnectionImplementation in the pool.

          Show
          shengjie min added a comment - In my case I am using HTablePool, my understanding is the following: so when the hbase query comes, if pool is avaiable: I get a PooledHtable instance, once I am done with query, I call HtablePool.PooledHtable.close(), it will return PooledHtable instance to the pool. if pool is out: I get a normal Htable instance, once I am done with query, I call Htable.close(), it will just release Htable instance. I have no clue why I ended up having closed HConnectionImplementation in the pool.
          Hide
          Adrian Muraru added a comment -

          I'm seeing a similar stack trace when using HTablePool:

          java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5c429be9 closed
                  at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38) ~[hbase-0.92.2-26.jar:0.92.2-26]
                  at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268) ~[hbase-0.92.2-26.jar:0.92.2-26]
                  at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198) ~[hbase-0.92.2-26.jar:0.92.2-26]
                  at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173) ~[hbase-0.92.2-26.jar:0.92.2-26]
          

          The issue is not with a closed HTable instance returned back to the pool.
          Instead, following my investigation it is caused by the shared HConnection instance being closed.
          Once you get into this state - the HTablePool#getTable() will always fail and never recover.

          A possible fix is to drop the HConnection used by HTablePool#getTable() in case of an IOException.
          Not trivial though as it is created based on a private Configuration instance in the pool

          this.connection = HConnectionManager.getConnection(conf)
          

          A more brutal way of escaping this state is to drop the pool altogether a create a new one (not really doable in all cases depending on the usage)

          I'll try to come up with a patch to clean up the faulty HConnection and keep the pool up.

          Show
          Adrian Muraru added a comment - I'm seeing a similar stack trace when using HTablePool: java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5c429be9 closed at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38) ~[hbase-0.92.2-26.jar:0.92.2-26] at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268) ~[hbase-0.92.2-26.jar:0.92.2-26] at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198) ~[hbase-0.92.2-26.jar:0.92.2-26] at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173) ~[hbase-0.92.2-26.jar:0.92.2-26] The issue is not with a closed HTable instance returned back to the pool. Instead, following my investigation it is caused by the shared HConnection instance being closed. Once you get into this state - the HTablePool#getTable() will always fail and never recover. A possible fix is to drop the HConnection used by HTablePool#getTable() in case of an IOException. Not trivial though as it is created based on a private Configuration instance in the pool this .connection = HConnectionManager.getConnection(conf) A more brutal way of escaping this state is to drop the pool altogether a create a new one (not really doable in all cases depending on the usage) I'll try to come up with a patch to clean up the faulty HConnection and keep the pool up.
          Hide
          shengjie min added a comment -

          Adrian Muraruwhat version you are using?

          Show
          shengjie min added a comment - Adrian Muraru what version you are using?
          Hide
          Adrian Muraru added a comment -

          shengjie min 0.92.2

          Show
          Adrian Muraru added a comment - shengjie min 0.92.2
          Hide
          Bryan Baugher added a comment -

          We have also ran into this issue specifically during major failure of the cluster (i.e. hmaster, zookeeper, regionservers are down).

          Adrian Muraru did you manage to find a good way to flush a bad table from your pool when the connection becomes closed?

          Show
          Bryan Baugher added a comment - We have also ran into this issue specifically during major failure of the cluster (i.e. hmaster, zookeeper, regionservers are down). Adrian Muraru did you manage to find a good way to flush a bad table from your pool when the connection becomes closed?
          Hide
          Igor Yurinok added a comment -

          We implemented our own HTableFactory and HTable which able to check whether connection closed:

          public class HumbleHTableFactory implements HTableInterfaceFactory {
          
              @Override
              public HTableInterface createHTableInterface(Configuration config, byte[] tableName) {
                  try {
                      return new HumbleHTable(config, tableName);
                  } catch (IOException ioe) {
                      throw new RuntimeException(ioe);
                  }
              }
          
              @Override
              public void releaseHTableInterface(HTableInterface table) {
                  try {
                      table.close();
                  } catch (IOException ioe) {
                      throw new RuntimeException(ioe);
                  }
              }
          
          }
          
          public class HumbleHTable extends HTable {
          
              private static final Log log = LogFactory.getLog(HumbleHTable.class);
          
              public HumbleHTable(String tableName) throws IOException {
                  super(tableName);
              }
          
              public HumbleHTable(byte[] tableName) throws IOException {
                  super(tableName);
              }
          
              public HumbleHTable(Configuration conf, String tableName) throws IOException {
                  super(conf, tableName);
              }
          
              public HumbleHTable(Configuration conf, byte[] tableName) throws IOException {
                  super(conf, tableName);
              }
          
              /**
               * Harmless clean-up - HConnection isn't touched. Only the executor pool is shut down
               *
               * @throws IOException
               */
              @Override
              public void close() throws IOException {
                  if (isClosed()) {
                      return;
                  }
                  flushCommits();
                  ExecutorService pool = getPool();
                  if (pool != null) {
                      pool.shutdown();
                  }
                  setClosed();
              }
          
              private boolean isClosed() {
                  try {
                      return getSuperField("closed").getBoolean(this);
                  } catch (Exception e) {
                      log.error(e);
                      return false;
                  }
              }
          
              private void setClosed() {
                  try {
                      getSuperField("closed").setBoolean(this, true);
                  } catch (Exception e) {
                      log.error(e);
                  }
              }
          
              private ExecutorService getPool() {
                  try {
                      return (ExecutorService) getSuperField("pool").get(this);
                  } catch (Exception e) {
                      log.error(e);
                      return null;
                  }
              }
          
              /**
               * Loads stuff from the parent class via reflection
               */
              private Field getSuperField(String fieldName) throws NoSuchFieldException {
                  Field field = HTable.class.getDeclaredField(fieldName);
                  field.setAccessible(true);
                  return field;
              }
          
          }
          
          Show
          Igor Yurinok added a comment - We implemented our own HTableFactory and HTable which able to check whether connection closed: public class HumbleHTableFactory implements HTableInterfaceFactory { @Override public HTableInterface createHTableInterface(Configuration config, byte [] tableName) { try { return new HumbleHTable(config, tableName); } catch (IOException ioe) { throw new RuntimeException(ioe); } } @Override public void releaseHTableInterface(HTableInterface table) { try { table.close(); } catch (IOException ioe) { throw new RuntimeException(ioe); } } } public class HumbleHTable extends HTable { private static final Log log = LogFactory.getLog(HumbleHTable.class); public HumbleHTable( String tableName) throws IOException { super (tableName); } public HumbleHTable( byte [] tableName) throws IOException { super (tableName); } public HumbleHTable(Configuration conf, String tableName) throws IOException { super (conf, tableName); } public HumbleHTable(Configuration conf, byte [] tableName) throws IOException { super (conf, tableName); } /** * Harmless clean-up - HConnection isn't touched. Only the executor pool is shut down * * @ throws IOException */ @Override public void close() throws IOException { if (isClosed()) { return ; } flushCommits(); ExecutorService pool = getPool(); if (pool != null ) { pool.shutdown(); } setClosed(); } private boolean isClosed() { try { return getSuperField( "closed" ).getBoolean( this ); } catch (Exception e) { log.error(e); return false ; } } private void setClosed() { try { getSuperField( "closed" ).setBoolean( this , true ); } catch (Exception e) { log.error(e); } } private ExecutorService getPool() { try { return (ExecutorService) getSuperField( "pool" ).get( this ); } catch (Exception e) { log.error(e); return null ; } } /** * Loads stuff from the parent class via reflection */ private Field getSuperField( String fieldName) throws NoSuchFieldException { Field field = HTable.class.getDeclaredField(fieldName); field.setAccessible( true ); return field; } }
          Hide
          stack added a comment -

          This seems like an ugly enough oversight on our part. Any chance of a patch to address it? Could we change HConnection Interface to make it easier getting a new one Adrian Muraru? (I like the name of your class Igor Yurinok)

          Show
          stack added a comment - This seems like an ugly enough oversight on our part. Any chance of a patch to address it? Could we change HConnection Interface to make it easier getting a new one Adrian Muraru ? (I like the name of your class Igor Yurinok )
          Hide
          Micah Whitacre added a comment -

          Igor, can you explain how you are detecting the connection is closed? Your isClosed() method only checks to see if the HTable boolean value has been closed which is independent of the HConnection.

          Additionally why do you avoid trying to close the HConnection by informing the HConnectionManager that the connection is stale?

          The last question is how does this solve the problem of evicting HTableInterface instances from the pool that are no longer valid? Altering the HTableInterfaceFactory + HTableInterface implementation only solves the problem of new instances needed to be created.

          Show
          Micah Whitacre added a comment - Igor, can you explain how you are detecting the connection is closed? Your isClosed() method only checks to see if the HTable boolean value has been closed which is independent of the HConnection. Additionally why do you avoid trying to close the HConnection by informing the HConnectionManager that the connection is stale? The last question is how does this solve the problem of evicting HTableInterface instances from the pool that are no longer valid? Altering the HTableInterfaceFactory + HTableInterface implementation only solves the problem of new instances needed to be created.
          Hide
          Adrian Muraru added a comment -

          There is an ongoing discussion on HBASE-6580 addressing this issue, a broader scope but would resolve the broken HConnection discussed here.
          The agreement there was to deprecate HTablePool and rely on lightweight (read cheap) HTable (HBASE-4805) retrieved from a new HConnection#getTable()

          I have a patch on HBASE-6580 that proposes a new HTableInterfaceFactory :https://issues.apache.org/jira/secure/attachment/12554094/HBASE-6580_v1.patch
          and can be used as a custom factory when creating the HTablePool

          org.apache.hadoop.hbase.client.HTablePool.HTablePool(Configuration config, int maxSize, HTableInterfaceFactory tableFactory)
          Show
          Adrian Muraru added a comment - There is an ongoing discussion on HBASE-6580 addressing this issue, a broader scope but would resolve the broken HConnection discussed here. The agreement there was to deprecate HTablePool and rely on lightweight (read cheap) HTable ( HBASE-4805 ) retrieved from a new HConnection#getTable() I have a patch on HBASE-6580 that proposes a new HTableInterfaceFactory : https://issues.apache.org/jira/secure/attachment/12554094/HBASE-6580_v1.patch and can be used as a custom factory when creating the HTablePool org.apache.hadoop.hbase.client.HTablePool.HTablePool(Configuration config, int maxSize, HTableInterfaceFactory tableFactory)

            People

            • Assignee:
              Unassigned
              Reporter:
              Igor Yurinok
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development