Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21657

PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% scan case.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0
    • Component/s: Performance
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      In HBASE-21657, I simplified the path of estimatedSerialiedSize() & estimatedSerialiedSizeOfCell() by moving the general getSerializedSize()
      and heapSize() from ExtendedCell to Cell interface. The patch also included some other improvments:

      1. For 99% of case, our cells has no tags, so let the HFileScannerImpl just return the NoTagsByteBufferKeyValue if no tags, which means we can save
         lots of cpu time when sending no tags cell to rpc because can just return the length instead of getting the serialize size by caculating offset/length
         of each fields(row/cf/cq..)
      2. Move the subclass's getSerializedSize implementation from ExtendedCell to their own class, which mean we did not need to call ExtendedCell's
         getSerialiedSize() firstly, then forward to subclass's getSerializedSize(withTags).
      3. Give a estimated result arraylist size for avoiding the frequent list extension when in a big scan, now we estimate the array size as min(scan.rows, 512).
         it's also help a lot.

      We gain almost ~40% throughput improvement in 100% scan case for branch-2 (cacheHitRatio~100%)[1], it's a good thing. While it's a incompatible change in
      some case, such as if the upstream user implemented their own Cells, although it's rare but can happen, then their compile will be error.
      Show
      In HBASE-21657 , I simplified the path of estimatedSerialiedSize() & estimatedSerialiedSizeOfCell() by moving the general getSerializedSize() and heapSize() from ExtendedCell to Cell interface. The patch also included some other improvments: 1. For 99% of case, our cells has no tags, so let the HFileScannerImpl just return the NoTagsByteBufferKeyValue if no tags, which means we can save    lots of cpu time when sending no tags cell to rpc because can just return the length instead of getting the serialize size by caculating offset/length    of each fields(row/cf/cq..) 2. Move the subclass's getSerializedSize implementation from ExtendedCell to their own class, which mean we did not need to call ExtendedCell's    getSerialiedSize() firstly, then forward to subclass's getSerializedSize(withTags). 3. Give a estimated result arraylist size for avoiding the frequent list extension when in a big scan, now we estimate the array size as min(scan.rows, 512).    it's also help a lot. We gain almost ~40% throughput improvement in 100% scan case for branch-2 (cacheHitRatio~100%)[1], it's a good thing. While it's a incompatible change in some case, such as if the upstream user implemented their own Cells, although it's rare but can happen, then their compile will be error.

      Description

      We are evaluating the performance of branch-2, and find that the throughput of scan in SSD cluster is almost the same as HDD cluster. so I made a FlameGraph on RS, and found that the PrivateCellUtil#estimatedSerializedSizeOf cost about 29% cpu, Obviously, it has been the bottleneck in 100% scan case.

      See theĀ hbase20-ssd-100-scan-traces.svg

      BTW, in our XiaoMi branch, we introduce a HRegion#updateReadRequestsByCapacityUnitPerSecond to sum up the size of cells (for metric monitor), so it seems the performance loss was amplified.

        Attachments

        1. debug-the-ByteBufferKeyValue.diff
          3 kB
          Zheng Hu
        2. HBase1.4.9-ssd-10000000-rows-flamegraph.svg
          619 kB
          Zheng Hu
        3. HBase1.4.9-ssd-10000000-rows-qps-latency.png
          146 kB
          Zheng Hu
        4. HBase2.0.4-patch-v2-ssd-10000000-rows.svg
          451 kB
          Zheng Hu
        5. HBase2.0.4-patch-v2-ssd-10000000-rows-qps-and-latency.png
          156 kB
          Zheng Hu
        6. HBase2.0.4-patch-v3-ssd-10000000-rows-flamegraph.svg
          756 kB
          Zheng Hu
        7. HBase2.0.4-patch-v3-ssd-10000000-rows-qps-and-latency.png
          153 kB
          Zheng Hu
        8. HBase2.0.4-patch-v4-ssd-10000000-rows-flamegraph.svg
          727 kB
          Zheng Hu
        9. HBase2.0.4-ssd-10000000-rows-flamegraph.svg
          288 kB
          Zheng Hu
        10. HBase2.0.4-ssd-10000000-rows-qps-latency.png
          148 kB
          Zheng Hu
        11. hbase2.0.4-ssd-scan-traces.2.svg
          469 kB
          Zheng Hu
        12. hbase2.0.4-ssd-scan-traces.svg
          261 kB
          Zheng Hu
        13. HBase2.0.4-without-patch-v2.png
          183 kB
          Zheng Hu
        14. HBase2.0.4-with-patch.v2.png
          186 kB
          Zheng Hu
        15. hbase20-ssd-100-scan-traces.svg
          922 kB
          Zheng Hu
        16. HBASE-21657.v1.patch
          6 kB
          Zheng Hu
        17. HBASE-21657.v2.patch
          11 kB
          Zheng Hu
        18. HBASE-21657.v3.patch
          22 kB
          Zheng Hu
        19. HBASE-21657.v3.patch
          17 kB
          Zheng Hu
        20. HBASE-21657.v4.patch
          25 kB
          Zheng Hu
        21. HBASE-21657.v5.patch
          33 kB
          Michael Stack
        22. HBASE-21657.v5.patch
          33 kB
          Zheng Hu
        23. HBASE-21657.v5.patch
          33 kB
          Zheng Hu
        24. HBASE-21657.v6.patch
          30 kB
          Zheng Hu
        25. HBASE-21657.v7.patch
          31 kB
          Zheng Hu
        26. image-2019-01-07-19-03-37-930.png
          72 kB
          Zheng Hu
        27. image-2019-01-07-19-03-55-577.png
          72 kB
          Zheng Hu
        28. overview-statstics-1.png
          71 kB
          Zheng Hu
        29. run.log
          60 kB
          Zheng Hu

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: