ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1162

consistent handling of jute.maxbuffer when attempting to read large zk "directories"

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.3.3
    • Fix Version/s: 3.5.0
    • Component/s: server
    • Labels:
      None

      Description

      Recently we encountered a sitaution where a zk directory got sucessfully populated with 250k elements. When our system attempted to read the znode dir, it failed because the contents of the dir exceeded the default 1mb jute.maxbuffer limit. There were a few odd things

      1) It seems odd that we could populate to be very large but could not read the listing
      2) The workaround was bumping up jute.maxbuffer on the client side setting.

      Would it make more sense to have it reject adding new znodes if it exceeds jute.maxbuffer?
      Alternately, would it make sense to have zk dir listing ignore the jute.maxbuffer setting?

        Issue Links

          Activity

          Jonathan Hsieh created issue -
          Hide
          Patrick Hunt added a comment -

          I would be nice to address this, it comes up somewhat frequently - I believe watch re-registration and such are effected by this as well.

          Perhaps we should enforce this when setting data for a znode, but otw allow it to exceed the max when reading?

          Show
          Patrick Hunt added a comment - I would be nice to address this, it comes up somewhat frequently - I believe watch re-registration and such are effected by this as well. Perhaps we should enforce this when setting data for a znode, but otw allow it to exceed the max when reading?
          Patrick Hunt made changes -
          Field Original Value New Value
          Fix Version/s 3.5.0 [ 12316644 ]
          Priority Major [ 3 ] Critical [ 2 ]
          Component/s server [ 12312382 ]
          Hide
          Jonathan Hsieh added a comment -

          Basically, I feel that to be consistent behavior-wise it should either:

          1) reject on write when dir becomes too big, keeping the current read constraint (ideally in zk, as opposed to the client)
          2) accept write like currently but allow the read to then succeed in this particular case.
          3) warn when writing when it gets too big, and then allow reads to succeed even if too big.

          Show
          Jonathan Hsieh added a comment - Basically, I feel that to be consistent behavior-wise it should either: 1) reject on write when dir becomes too big, keeping the current read constraint (ideally in zk, as opposed to the client) 2) accept write like currently but allow the read to then succeed in this particular case. 3) warn when writing when it gets too big, and then allow reads to succeed even if too big.
          Patrick Hunt made changes -
          Link This issue breaks HBASE-4246 [ HBASE-4246 ]
          Hide
          Todd Lipcon added a comment -

          Does ZK use any checksums/magic numbers in its wire protocol? Typically I see limits like this on packet framing lengths so that, if you connect and send garbage to the IPC port, it won't try to allocate 4GB and crash. Adding a magic number before the length or a simple checksum would prevent the "garbage being interpreted as a length" issue, while still allowing large responses.

          Show
          Todd Lipcon added a comment - Does ZK use any checksums/magic numbers in its wire protocol? Typically I see limits like this on packet framing lengths so that, if you connect and send garbage to the IPC port, it won't try to allocate 4GB and crash. Adding a magic number before the length or a simple checksum would prevent the "garbage being interpreted as a length" issue, while still allowing large responses.
          Patrick Hunt made changes -
          Link This issue breaks ZOOKEEPER-706 [ ZOOKEEPER-706 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Jonathan Hsieh
            • Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development