Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14978

Don't allow Multi to retain too many blocks

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.2.0, 1.3.0, 2.0.0
    • 1.2.0, 1.3.0, 2.0.0
    • io, IPC/RPC, regionserver
    • None
    • Hide
      Limiting the amount of memory resident for any one request allows the server to handle concurrent requests smoothly. To this end we added the ability to limit the size of responses to a multi request. That worked well however it correctly represent the amount of memory resident. So this issue adds on a an approximation of the number of blocks held for a request.

      All clients before 1.2.0 will not get this multi request chunking based upon blocks kept. All clients 1.2.0 and after will.
      Show
      Limiting the amount of memory resident for any one request allows the server to handle concurrent requests smoothly. To this end we added the ability to limit the size of responses to a multi request. That worked well however it correctly represent the amount of memory resident. So this issue adds on a an approximation of the number of blocks held for a request. All clients before 1.2.0 will not get this multi request chunking based upon blocks kept. All clients 1.2.0 and after will.

    Description

      Scans and Multi's have limits on the total size of cells that can be returned. However if those requests are not all pointing at the same blocks then the KeyValues can keep alive a lot more data than their size.

      Take the following example:

      A multi with a list of 10000 gets to a fat row. Each column being returned in in a different block. Each column is small 32 bytes or so.

      So the total cell size will be 32 * 10000 = ~320kb. However if each block is 128k then total retained heap size will be almost 2gigs.

      Attachments

        1. HBASE-14978.patch
          13 kB
          Elliott Neil Clark
        2. HBASE-14978-branch-1.2.patch
          14 kB
          Elliott Neil Clark
        3. HBASE-14978-branch-1.2.patch
          16 kB
          Elliott Neil Clark
        4. HBASE-14978-branch-1.2.patch
          17 kB
          Elliott Neil Clark
        5. HBASE-14978-branch-1.2.patch
          16 kB
          Elliott Neil Clark
        6. HBASE-14978-v1.patch
          14 kB
          Elliott Neil Clark
        7. HBASE-14978-v2.patch
          14 kB
          Elliott Neil Clark
        8. HBASE-14978-v3.patch
          14 kB
          Elliott Neil Clark
        9. HBASE-14978-v4.patch
          14 kB
          Elliott Neil Clark

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            eclark Elliott Neil Clark
            eclark Elliott Neil Clark
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment