Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-2375

Refactor BlockOutputStream to allow flexible buffering

    XMLWordPrintableJSON

    Details

      Description

      In HDDS-2331, we found that Ozone client allocates a ByteBuffer with chunk size (e.g. 16MB ) to store data, unregarded the actual data size. The ByteBuffer will create a byte[] with chunk size. When the ByteBuffer is wrapped to a ByteString the byte[] remains in the ByteString.

      As a result, when the actual data size is small (e.g. 1MB), a lot of memory spaces (15MB) are wasted.

      In this JIRA, we refactor BlockOutputStream so that the buffering becomes more flexible. In a later JIRA (HDDS-2386), we implement chunk buffer using a list of smaller buffers which are allocated only if needed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                szetszwo Tsz-wo Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m