HBase
  1. HBase
  2. HBASE-5190

Limit the IPC queue size based on calls' payload size

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.5
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Each region server can only have 1GB worth of calls's payload in flight and will refuse new calls when above that limit (those get automatically retried on the client side). Also, ipc.server.max.queue.size is now called ipc.server.max.callqueue.length

      Description

      Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions:

      • Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception.
      • Same accounting but instead block the call when it comes in until space is made available.
      • Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue.
      1. HBASE-5190-v3.patch
        7 kB
        Jean-Daniel Cryans
      2. HBASE-5190-v2.patch
        6 kB
        Jean-Daniel Cryans
      3. HBASE-5190.patch
        6 kB
        Jean-Daniel Cryans
      4. 5190.addendum
        2 kB
        Ted Yu

        Activity

        Lars Hofhansl made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.94.0 [ 12316419 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Jean-Daniel Cryans made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Ted Yu made changes -
        Attachment 5190.addendum [ 12519871 ]
        Ted Yu made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Jean-Daniel Cryans made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Incompatible change,Reviewed [ 10342,10343 ]
        Release Note Each region server can only have 1GB worth of calls's payload in flight and will refuse new calls when above that limit (those get automatically retried on the client side). Also, ipc.server.max.queue.size is now called ipc.server.max.callqueue.length
        Fix Version/s 0.96.0 [ 12320040 ]
        Resolution Fixed [ 1 ]
        Jean-Daniel Cryans made changes -
        Attachment HBASE-5190-v3.patch [ 12519540 ]
        Jean-Daniel Cryans made changes -
        Attachment HBASE-5190-v2.patch [ 12519520 ]
        Jean-Daniel Cryans made changes -
        Attachment HBASE-5190.patch [ 12510553 ]
        Jean-Daniel Cryans made changes -
        Field Original Value New Value
        Assignee Jean-Daniel Cryans [ jdcryans ]
        Jean-Daniel Cryans created issue -

          People

          • Assignee:
            Jean-Daniel Cryans
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development