Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1911

MergeManager's unconditionalReserve() should check for memory limits before allocating memory to IntermediateMemoryToMemoryMerger

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.7.1, 0.8.3
    • None
    • None
    • Reviewed

    Description

      Currently, IntermediateMemoryToMemoryMerger invokes unconditionalReserve() to get the memory needed for intermediate mem-to-mem merging. It could potentially cause issue in the following scenario

      1. tez.runtime.io.sort.factor set to 100 and assume shuffled data (e.g 60% memory occupied) haven't reached TEZ_RUNTIME_SHUFFLE_MERGE_PERCENT_DEFAULT
      2. Assume that it reaches the sort.factor threshold before reaching merge threshold. This would kick in IntermediateMemoryToMemoryMerger.

      In IntermediateMemoryToMemoryMerger, it would try to allocate additional 60% without any boundary checks. This could lead to OOM depending on the tez.runtime.io.sort.mb setting.

      Attachments

        1. TEZ-1911.addendum.findbugs.patch
          2 kB
          Rajesh Balamohan
        2. TEZ-1911.3.patch
          15 kB
          Rajesh Balamohan
        3. TEZ-1911.2.patch
          14 kB
          Rajesh Balamohan
        4. TEZ-1911.1.patch
          15 kB
          Rajesh Balamohan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rajesh.balamohan Rajesh Balamohan
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment