Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21657

PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% scan case.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0
    • Component/s: Performance
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      In HBASE-21657, I simplified the path of estimatedSerialiedSize() & estimatedSerialiedSizeOfCell() by moving the general getSerializedSize()
      and heapSize() from ExtendedCell to Cell interface. The patch also included some other improvments:

      1. For 99% of case, our cells has no tags, so let the HFileScannerImpl just return the NoTagsByteBufferKeyValue if no tags, which means we can save
         lots of cpu time when sending no tags cell to rpc because can just return the length instead of getting the serialize size by caculating offset/length
         of each fields(row/cf/cq..)
      2. Move the subclass's getSerializedSize implementation from ExtendedCell to their own class, which mean we did not need to call ExtendedCell's
         getSerialiedSize() firstly, then forward to subclass's getSerializedSize(withTags).
      3. Give a estimated result arraylist size for avoiding the frequent list extension when in a big scan, now we estimate the array size as min(scan.rows, 512).
         it's also help a lot.

      We gain almost ~40% throughput improvement in 100% scan case for branch-2 (cacheHitRatio~100%)[1], it's a good thing. While it's a incompatible change in
      some case, such as if the upstream user implemented their own Cells, although it's rare but can happen, then their compile will be error.
      Show
      In HBASE-21657 , I simplified the path of estimatedSerialiedSize() & estimatedSerialiedSizeOfCell() by moving the general getSerializedSize() and heapSize() from ExtendedCell to Cell interface. The patch also included some other improvments: 1. For 99% of case, our cells has no tags, so let the HFileScannerImpl just return the NoTagsByteBufferKeyValue if no tags, which means we can save    lots of cpu time when sending no tags cell to rpc because can just return the length instead of getting the serialize size by caculating offset/length    of each fields(row/cf/cq..) 2. Move the subclass's getSerializedSize implementation from ExtendedCell to their own class, which mean we did not need to call ExtendedCell's    getSerialiedSize() firstly, then forward to subclass's getSerializedSize(withTags). 3. Give a estimated result arraylist size for avoiding the frequent list extension when in a big scan, now we estimate the array size as min(scan.rows, 512).    it's also help a lot. We gain almost ~40% throughput improvement in 100% scan case for branch-2 (cacheHitRatio~100%)[1], it's a good thing. While it's a incompatible change in some case, such as if the upstream user implemented their own Cells, although it's rare but can happen, then their compile will be error.

      Description

      We are evaluating the performance of branch-2, and find that the throughput of scan in SSD cluster is almost the same as HDD cluster. so I made a FlameGraph on RS, and found that the PrivateCellUtil#estimatedSerializedSizeOf cost about 29% cpu, Obviously, it has been the bottleneck in 100% scan case.

      See theĀ hbase20-ssd-100-scan-traces.svg

      BTW, in our XiaoMi branch, we introduce a HRegion#updateReadRequestsByCapacityUnitPerSecond to sum up the size of cells (for metric monitor), so it seems the performance loss was amplified.

        Attachments

        1. HBASE-21657.v7.patch
          31 kB
          Zheng Hu
        2. HBASE-21657.v6.patch
          30 kB
          Zheng Hu
        3. HBASE-21657.v5.patch
          33 kB
          stack
        4. HBASE-21657.v5.patch
          33 kB
          Zheng Hu
        5. HBASE-21657.v5.patch
          33 kB
          Zheng Hu
        6. HBase2.0.4-patch-v4-ssd-10000000-rows-flamegraph.svg
          727 kB
          Zheng Hu
        7. HBASE-21657.v4.patch
          25 kB
          Zheng Hu
        8. debug-the-ByteBufferKeyValue.diff
          3 kB
          Zheng Hu
        9. image-2019-01-07-19-03-55-577.png
          72 kB
          Zheng Hu
        10. image-2019-01-07-19-03-37-930.png
          72 kB
          Zheng Hu
        11. HBase2.0.4-patch-v3-ssd-10000000-rows-flamegraph.svg
          756 kB
          Zheng Hu
        12. HBase2.0.4-patch-v3-ssd-10000000-rows-qps-and-latency.png
          153 kB
          Zheng Hu
        13. run.log
          60 kB
          Zheng Hu
        14. HBASE-21657.v3.patch
          22 kB
          Zheng Hu
        15. HBASE-21657.v3.patch
          17 kB
          Zheng Hu
        16. overview-statstics-1.png
          71 kB
          Zheng Hu
        17. HBase1.4.9-ssd-10000000-rows-flamegraph.svg
          619 kB
          Zheng Hu
        18. HBase2.0.4-patch-v2-ssd-10000000-rows.svg
          451 kB
          Zheng Hu
        19. HBase2.0.4-ssd-10000000-rows-flamegraph.svg
          288 kB
          Zheng Hu
        20. HBase2.0.4-patch-v2-ssd-10000000-rows-qps-and-latency.png
          156 kB
          Zheng Hu
        21. HBase1.4.9-ssd-10000000-rows-qps-latency.png
          146 kB
          Zheng Hu
        22. HBase2.0.4-ssd-10000000-rows-qps-latency.png
          148 kB
          Zheng Hu
        23. HBase2.0.4-without-patch-v2.png
          183 kB
          Zheng Hu
        24. HBase2.0.4-with-patch.v2.png
          186 kB
          Zheng Hu
        25. hbase2.0.4-ssd-scan-traces.2.svg
          469 kB
          Zheng Hu
        26. HBASE-21657.v2.patch
          11 kB
          Zheng Hu
        27. hbase2.0.4-ssd-scan-traces.svg
          261 kB
          Zheng Hu
        28. HBASE-21657.v1.patch
          6 kB
          Zheng Hu
        29. hbase20-ssd-100-scan-traces.svg
          922 kB
          Zheng Hu

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: