Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34975 FLIP-427: ForSt - Disaggregated State Store
  3. FLINK-36526

Optimize the overhead of writing with direct buffer in ForSt

    XMLWordPrintableJSON

Details

    Description

      Currently, the ForSt gives a direct buffer to ByteBufferWritableFSDataOutputStream, where the data will be written one byte by byte. According our perf, the statistics of hadoop based fs will be updated once for each byte, which takes a lot of CPU. Below is a flamegraph, where the statistics part is marked as purple (taking 8.14% of the overall CPU).

       

      It might be better to copy to a heap buffer before invoking write.

      Attachments

        1. image-2024-10-14-15-52-41-457.png
          2.89 MB
          Zakelly Lan

        Issue Links

          Activity

            People

              zakelly Zakelly Lan
              zakelly Zakelly Lan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: