Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
  3. HBASE-22663

The HeapAllocationRatio in WebUI is not accurate because all of the heap allocation will happen in another separated allocator named HEAP

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We use the following method to calculate the heap allocation ratio:

      public double getHeapAllocationRatio() {
          long heapAllocBytes = heapAllocationBytes.sum(), poolAllocBytes = poolAllocationBytes.sum();
          double heapDelta = heapAllocBytes - lastHeapAllocationBytes;
          double poolDelta = poolAllocBytes - lastPoolAllocationBytes;
          lastHeapAllocationBytes = heapAllocBytes;
          lastPoolAllocationBytes = poolAllocBytes;
          if (Math.abs(heapDelta + poolDelta) < 1e-3) {
            return 0.0;
          }
          return heapDelta / (heapDelta + poolDelta);
        }
      

      While we also defined a static allocator instance, named HEAP:

        // The on-heap allocator is mostly used for testing, but also some non-test usage, such as
        // scanning snapshot, we won't have an RpcServer to initialize the allocator, so just use the
        // default heap allocator, it will just allocate ByteBuffers from heap but wrapped by an ByteBuff.
        public static final ByteBuffAllocator HEAP = ByteBuffAllocator.createOnHeap();
      

      Almost all of the heap allocation happen in the HEAP, which means the heap allocation ratio caculated from the rpc ByteBuffAllocator will be not accurate.

      Will prepare a patch to address this minor bug.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: