Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25905

BlockManager should expose getRemoteManagedBuffer to avoid creating bytebuffers

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:

      Description

      The block manager currently only lets you get a handle on remote data as a ChunkedByteBuffer. But with remote reads of cached data, you really only need an input stream view of the data, which is already available with the ManagedBuffer which is fetched. By forcing conversion to a ChunkedByteBuffer, we end up using more memory than necessary.

        Attachments

          Activity

            People

            • Assignee:
              wypoon Wing Yew Poon
              Reporter:
              irashid Imran Rashid
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: