Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
  3. HBASE-22532

There's still too much cpu wasting on validating checksum even if buffer.size=65KB

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      After disabled the block cache, and with the following config:

          # Disable the block cache
          hfile.block.cache.size=0
          hbase.ipc.server.allocator.buffer.size=66560
          hbase.ipc.server.reservoir.minimal.allocating.size=0
      

      The ByteBuff for block should be expected to be a SingleByteBuff, which will use the hadoop native lib to validate the checksum, while in the cpu flame graph async-prof-pid-27827-cpu-3.svg, we can still see that about 32% CPU wasted on PureJavaCrc32#update, which means it's not using the faster hadoop native lib.

        Attachments

        1. async-prof-pid-64695-cpu-1.svg
          359 kB
          Zheng Hu
        2. async-prof-pid-27827-cpu-3.svg
          348 kB
          Zheng Hu

          Activity

            People

            • Assignee:
              openinx Zheng Hu
              Reporter:
              openinx Zheng Hu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: