Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-17818

Tez & Hive: Memory management should use scaled down Xmx for sizing buffers

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • stacks
    • None

    Description

      The defaults in Ambari assume 80% of the YARN container size is allocate to the user heap (Xmx).

      On top of this particular restriction, the allocation blocks are divided into Eden/Survivor/Tenured - a single allocation spanning 40% of the heap might fail for tez runtime.io.sort.mb.

      Scale down the Tez sort size and map join by 0.8x to match Xmx.

      Attachments

        1. AMBARI-17818.1.patch
          7 kB
          Gopal Vijayaraghavan
        2. AMBARI-17818.2.patch
          7 kB
          Gopal Vijayaraghavan

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gopalv Gopal Vijayaraghavan
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment