Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-4306

CloseSessionTxn contains too many ephemal nodes cause cluster crash

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.6.2
    • Fix Version/s: None
    • Component/s: None

      Description

      We took a test about how many ephemal nodes can client create under one parent node with defalut configuration. The test caused cluster crash at last, exception stack trace like this.

      follower:

      leader:

      It seems that leader sent a too large txn packet to followers. When follower try to deserialize the txn, it found the txn length out of its buffer size(default 1MB+1MB, jute.maxbuffer + jute.maxbuffer.extrasize). That causes followers crashed, and then, leader found there was no sufficient followers synced, so leader shutdown later. When leader shutdown, it called zkDb.fastForwardDataBase() , and leader found the txn read from txnlog out of its buffer size, so it crashed too.

      After the servers crashed, they try to restart the quorum. But they would not success because the last txn is too large. We lose the log at that moment, but the stack trace is same as this one.

       

      Root Cause

      We use org.apache.zookeeper.server.LogFormatter(-Djute.maxbuffer=74827780) visualize this log and found this. So closeSessionTxn contains all ephemal nodes with absolute path. We know we will get a large getChildren respose if we create too many children nodes under one parent node, that is limited by jute.maxbuffer of client. If we create plenty of ephemal nodes under different parent nodes with one session, it may not cause out of buffer of client, but when the session close without delete these node first, it probably cause cluster crash.

      Is it a bug or just a unspecified feature?If it just so, how should we judge the upper limit of creating nodes? 

       

        Attachments

        1. cs.jpg
          553 kB
          Lin Changrui
        2. f.jpg
          55 kB
          Lin Changrui
        3. l1.png
          9 kB
          Lin Changrui
        4. l2.jpg
          108 kB
          Lin Changrui
        5. r.jpg
          317 kB
          Lin Changrui

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Changrui Lin Lin Changrui
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h