Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6721

mapreduce.reduce.shuffle.memory.limit.percent=0.0 should be legal to enforce shuffle to disk

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: mrv2, task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We are potentially hitting an in-memory-shuffle-related reservation starvation resembling MAPREDUCE-6445. To work it around, we wanted to disable in memory shuffle via mapreduce.reduce.shuffle.memory.limit.percent=0.0 that turned out to be disallowed by the current logic. So we had to resort to another small float value such as 0.0001. However, zero is more logical imo.

        Attachments

        1. MAPREDUCE-6721.002.patch
          4 kB
          Gera Shegalov
        2. MAPREDUCE-6721.001.patch
          4 kB
          Gera Shegalov

          Issue Links

            Activity

              People

              • Assignee:
                jira.shegalov Gera Shegalov
                Reporter:
                jira.shegalov Gera Shegalov
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: