Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5027

Shuffle does not limit number of outstanding connections

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 0.23.5
    • Fix Version/s: 3.0.0, 0.23.7, 2.1.0-beta
    • Component/s: None
    • Labels:
      None

      Description

      The ShuffleHandler does not have any configurable limits to the number of outstanding connections allowed. Therefore a node with many map outputs and many reducers in the cluster trying to fetch those outputs can exhaust a nodemanager out of file descriptors.

      1. MAPREDUCE-5027.patch
        10 kB
        Robert Parker
      2. MAPREDUCE-5027.patch
        10 kB
        Robert Parker
      3. MAPREDUCE-5027-2.patch
        11 kB
        Robert Parker
      4. MAPREDUCE-5027-3.patch
        10 kB
        Robert Parker
      5. MAPREDUCE-5027-4.patch
        10 kB
        Robert Parker
      6. MAPREDUCE-5027-b023.patch
        10 kB
        Robert Parker
      7. MAPREDUCE-5027-b023-2.patch
        11 kB
        Robert Parker

        Activity

        Jason Lowe created issue -
        Robert Parker made changes -
        Field Original Value New Value
        Assignee Robert Parker [ robsparker ]
        Robert Parker made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571028 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571063 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571085 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571028 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571063 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571092 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571085 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027.patch [ 12571124 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027-2.patch [ 12571527 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027-3.patch [ 12571838 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027-b023.patch [ 12571895 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027-4.patch [ 12572160 ]
        Attachment MAPREDUCE-5027-b023-2.patch [ 12572161 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027-4.patch [ 12572165 ]
        Robert Parker made changes -
        Attachment MAPREDUCE-5027-4.patch [ 12572160 ]
        Jonathan Eagles made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Fix Version/s 0.23.7 [ 12323954 ]
        Fix Version/s 2.0.4-beta [ 12324032 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Robert Parker
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development