Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-7593 Supporting HSync and lease recovery
  3. HDDS-8047

[hsync] Make Putblock performance acceptable

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Implemented
    • None
    • None
    • None

    Description

      In a profile result, we found that the client PutBlock request has lots of redundancy, especially the BlockData structure. The data structure contains the metadata of every chunk in the block, which is not required because only the last two chunks matters.

      This is especially critical for small and frequent puts, such as hsync API called by HBase WAL files. We found that at least 60% of client CPU cycle during hsync call can be attributed to this problem, and potentially a huge impact to the latency.

      Attachments

        1. after_hdds_8047.html
          846 kB
          Wei-Chiu Chuang
        2. hbase_rs.html
          502 kB
          Wei-Chiu Chuang
        3. image-2023-04-19-15-11-23-576.png
          334 kB
          ChenXi
        4. PutBlock metadata optimization.pdf
          61 kB
          Wei-Chiu Chuang
        5. Redesign PutBlock ChunkList.pdf
          99 kB
          Wei-Chiu Chuang
        6. Support Incremental ChunkList in PutBlock requests.pdf
          145 kB
          Wei-Chiu Chuang

        Issue Links

          contains

          Sub-task - The sub-task of the issue HDDS-9750 [hsync] Make Putblock performance acceptable - Skeleton code

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDDS-9751 [hsync] Make Putblock performance acceptable - DataNode side

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDDS-9752 [hsync] Make Putblock performance acceptable - Client side

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDDS-9753 [hsync] Make Putblock performance acceptable - Tool cleanup, guardrails

          • Major - Major loss of function.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HDDS-7593 Supporting HSync and lease recovery

          • Major - Major loss of function.
          • Resolved
          is required by

          Sub-task - The sub-task of the issue HDDS-8769 [hsync] disk usage thread aborts if ratis log rolls very quickly

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link [Design Doc] Redesign PutBlock ChunkList

          Web Link GitHub Pull Request #4628

          Web Link GitHub Pull Request #5628

          Activity

            People

              weichiu Wei-Chiu Chuang
              weichiu Wei-Chiu Chuang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: