Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21029

Miscount of memstore's heap/offheap size if same cell was put



    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.0.1
    • Fix Version/s: 2.1.1, 2.0.2
    • Component/s: None
    • Labels:


      We are now using memstore.heapSize() + memstore.offheapSize() to decide whether a flush is needed. But, if a same cell was put again in memstore, only the memstore's dataSize will be increased, the heap/offheap size won't. We encountered the problem that a user was putting the same kv again and again, but the memstore won't flush since the heap size was not counted properly. The RS was killed by system since not enough memory in the end.

      Actually, if MSLAB is used, the heap/offheap will increase no matter the cell is added or not. IIRC, memstore's heap/offheap size should always bigger than data size. We introduced heap/offheap size besides data size in order to reflect memory footprint more precisely.

          // If there's already a same cell in the CellSet and we are using MSLAB, we must count in the
          // MSLAB allocation size as well, or else there will be memory leak (occupied heap size larger
          // than the counted number)
          if (succ || mslabUsed) {
            cellSize = getCellLength(cellToAdd);
          // heap/offheap size is changed only if the cell is truly added in the cellSet
          long heapSize = heapSizeChange(cellToAdd, succ);
          long offHeapSize = offHeapSizeChange(cellToAdd, succ);


        1. HBASE-21029.branch-2.0.001.patch
          3 kB
          Allan Yang
        2. HBASE-21029.branch-2.0.002.patch
          3 kB
          Allan Yang

          Issue Links



              • Assignee:
                allan163 Allan Yang
                allan163 Allan Yang
              • Votes:
                0 Vote for this issue
                7 Start watching this issue


                • Created: