Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5027

Shuffle does not limit number of outstanding connections

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 0.23.5
    • Fix Version/s: 0.23.7, 2.1.0-beta
    • Component/s: None
    • Labels:
      None

      Description

      The ShuffleHandler does not have any configurable limits to the number of outstanding connections allowed. Therefore a node with many map outputs and many reducers in the cluster trying to fetch those outputs can exhaust a nodemanager out of file descriptors.

        Attachments

        1. MAPREDUCE-5027.patch
          10 kB
          Robert Parker
        2. MAPREDUCE-5027.patch
          10 kB
          Robert Parker
        3. MAPREDUCE-5027-2.patch
          11 kB
          Robert Parker
        4. MAPREDUCE-5027-3.patch
          10 kB
          Robert Parker
        5. MAPREDUCE-5027-4.patch
          10 kB
          Robert Parker
        6. MAPREDUCE-5027-b023.patch
          10 kB
          Robert Parker
        7. MAPREDUCE-5027-b023-2.patch
          11 kB
          Robert Parker

          Activity

            People

            • Assignee:
              robsparker Robert Parker
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: