HBase
  1. HBase
  2. HBASE-2669

HCM.shutdownHook causes data loss with hbase.client.write.buffer != 0

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.90.0
    • Component/s: Client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      data loss

      Description

      In my application I set hbase.client.write.buffer to a reasonably small value (roughly 64 edits) in order to try to batch a few Put together before talking to HBase. When my application does a graceful shutdown, I call HTable#flushCommits in order to flush any pending change to HBase. I want to do the same thing when I get a SIGTERM by using Runtime#addShutdownHook but this is impossible since HConnectionManager already registers a shutdown hook that invokes HConnectionManager#deleteAllConnections. This static method closes all the connections to HBase and then all connections to ZooKeeper. Because all shutdown hooks run in parallel, my hook will attempt to flush edits while connections are getting closed.

      There is no way to guarantee the order in which the hooks will execute, so I propose that we remove the hook in the HCM altogether and provide some user-visible API they call in their own hook after they're done flushing their stuff, if they really want to do a graceful shutdown. I expect that a lot of users won't use a hook though, otherwise this issue would have cropped up already. For those users, connections won't get "gracefully" terminated, but I don't think that would be a problem since the underlying TCP socket will get closed by the OS anyway, so things like ZooKeeper and such should realize that the connection has been terminated and assume the client is gone, and do the necessary clean-up on their side.

      An alternate fix would be to leave the hook in place by default but keep a reference to it and add a user-visible API to be able to un-register the hook. I find this ugly.

      Thoughts?

      1. 2669.txt
        4 kB
        stack
      2. 2669-v2.txt
        24 kB
        stack

        Issue Links

          Activity

          stack made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          stack made changes -
          Attachment 2669-v2.txt [ 12457494 ]
          stack made changes -
          Assignee Benoit Sigoure [ tsuna ] stack [ stack ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          stack made changes -
          Link This issue is related to HBASE-2952 [ HBASE-2952 ]
          stack made changes -
          Attachment 2669.txt [ 12450726 ]
          stack made changes -
          Field Original Value New Value
          Fix Version/s 0.21.0 [ 12313607 ]
          Fix Version/s 0.20.5 [ 12314800 ]
          Benoit Sigoure created issue -

            People

            • Assignee:
              stack
              Reporter:
              Benoit Sigoure
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development