Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-2952

HConnectionManager's shutdown hook interferes with client's operations


    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: None
    • Component/s: Client
    • Labels:


      My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.

      For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.

      The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.

      My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.

      Signalled. Exiting ...
      2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
      flushing after 7899
      2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
      2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
      2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
      2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
      2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
      2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
      2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got
      2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.


      It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.


          Issue Links



              • Assignee:
                khemani Prakash Khemani
              • Votes:
                0 Vote for this issue
                0 Start watching this issue


                • Created: