Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19614 Further optimization of sort-merge based blocking shuffle
  3. FLINK-20740

Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)

Agile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Currently, sort-merge blocking shuffle uses some unmanaged memory for data writing and reading, which means users must increase the size of direct memory, otherwise, one may encounter direct memory OOM error, which is really bad for usability.

      As discussed in the mailing list, a separated buffer pool will be introduced to be used by the shuffle reading phase, this buffer pool is part of Flink framework and has different life cycle with the tasks.

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              kevin.cyj Yingjie Cao
              Reporter:
              kevin.cyj Yingjie Cao

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment