Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5027

Shuffle does not limit number of outstanding connections

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.3-alpha, 0.23.5
    • 0.23.7, 2.1.0-beta
    • None
    • None

    Description

      The ShuffleHandler does not have any configurable limits to the number of outstanding connections allowed. Therefore a node with many map outputs and many reducers in the cluster trying to fetch those outputs can exhaust a nodemanager out of file descriptors.

      Attachments

        1. MAPREDUCE-5027-b023-2.patch
          11 kB
          Robert Parker
        2. MAPREDUCE-5027-b023.patch
          10 kB
          Robert Parker
        3. MAPREDUCE-5027-4.patch
          10 kB
          Robert Parker
        4. MAPREDUCE-5027-3.patch
          10 kB
          Robert Parker
        5. MAPREDUCE-5027-2.patch
          11 kB
          Robert Parker
        6. MAPREDUCE-5027.patch
          10 kB
          Robert Parker
        7. MAPREDUCE-5027.patch
          10 kB
          Robert Parker

        Activity

          People

            robsparker Robert Parker
            jlowe Jason Darrell Lowe
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: