1. HBase
  2. HBASE-1316

ZooKeeper: use native threads to avoid GC stalls (JNI integration)


    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: 0.20.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      From Joey Echeverria up on hbase-users@:

      We've used zookeeper in a write-heavy project we've been working on and experienced issues similar to what you described. After several days of debugging, we discovered that our issue was garbage collection. There was no way to guarantee we wouldn't have long pauses especially since our environment was the worst case for garbage collection, millions of tiny, short lived objects. I suspect HBase sees similar work loads frequently, if it's not constantly. With anything shorter than a 30 second session time out, we got session expiration events extremely frequently. We needed to use 60 seconds for any real confidence that an ephemeral node disappearing meant something was unavailable.

      We really wanted quick recovery so we ended up writing a light-weight wrapper around the C API and used swig to auto-generate a JNI interface. It's not perfect, but since we switched to this method we've never seen a session expiration event and ephemeral nodes only disappear when there are network issues or a machine/process goes down.

      I don't know if it's worth doing the same kind of thing for HBase as it adds some "unnecessary" native code, but it's a solution that I found works.

      1. zk_wrapper.tar.gz
        1 kB
        Joey Echeverria
      2. zookeeper-native-headers.tgz
        15 kB
        Joey Echeverria
      3. zookeeper-native-Linux-amd64-64.tgz
        264 kB
        Joey Echeverria
      4. HBASE-1316-1.patch
        25 kB
        Joey Echeverria
      5. HBASE-1316-2.patch
        37 kB
        Joey Echeverria

        Issue Links


          Andrew Purtell made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Joey Echeverria [ fwiffo ]
          Resolution Later [ 7 ]
          Nicolas Liochon made changes -
          Link This issue relates to HBASE-5843 [ HBASE-5843 ]
          Joey Echeverria made changes -
          Assignee Berk D. Demir [ bdd ] Joey Echeverria [ fwiffo ]
          Joey Echeverria made changes -
          Attachment HBASE-1316-2.patch [ 12482984 ]
          Joey Echeverria made changes -
          Attachment zookeeper-native-headers.tgz [ 12479972 ]
          Attachment zookeeper-native-Linux-amd64-64.tgz [ 12479973 ]
          Attachment HBASE-1316-1.patch [ 12479974 ]
          Todd Lipcon made changes -
          Link This issue depends upon ZOOKEEPER-829 [ ZOOKEEPER-829 ]
          Berk D. Demir made changes -
          Assignee Nitay Joffe [ nitay ] Berk D. Demir [ bdd ]
          Nitay Joffe made changes -
          Assignee Nitay Joffe [ nitay ]
          Joey Echeverria made changes -
          Field Original Value New Value
          Attachment zk_wrapper.tar.gz [ 12405580 ]
          Andrew Purtell created issue -


            • Assignee:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              8 Start watching this issue


              • Created: