Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3490

RMContainerAllocator counts failed maps towards Reduce ramp up

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed MapReduce AM to count failed maps also towards Reduce ramp up.

      Description

      The RMContainerAllocator does not differentiate between failed and successful maps while calculating whether reduce tasks are ready to launch. Failed tasks are also counted towards total completed tasks.
      Example. 4 failed maps, 10 total maps. Map%complete = 4/14 * 100 instead of being 0.

      1. MAPREDUCE-3490.patch
        13 kB
        Arun C Murthy
      2. MAPREDUCE-3490.patch
        19 kB
        Arun C Murthy
      3. MAPREDUCE-3490.patch
        19 kB
        Arun C Murthy
      4. MAPREDUCE-3490.patch
        15 kB
        Arun C Murthy
      5. MAPREDUCE-3490.patch
        12 kB
        Arun C Murthy
      6. MAPREDUCE-3490.patch
        11 kB
        Arun C Murthy
      7. MAPREDUCE-3490.patch
        7 kB
        Arun C Murthy
      8. MR-3490-alternate.patch
        3 kB
        Sharad Agarwal
      9. MR-3490-alternate1.patch
        5 kB
        Sharad Agarwal

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Fixed MapReduce AM to count failed maps also towards Reduce ramp up.
        Fix Version/s 0.23.1 [ 12318883 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12509362 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12509178 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Arun C Murthy [ acmurthy ] Sharad Agarwal [ sharadag ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12509171 ]
        Sharad Agarwal made changes -
        Attachment MR-3490-alternate1.patch [ 12508753 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Sharad Agarwal made changes -
        Attachment MR-3490-alternate.patch [ 12508387 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12508338 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12508329 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12508327 ]
        Arun C Murthy made changes -
        Assignee Arun C Murthy [ acmurthy ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3490.patch [ 12508276 ]
        Arun C Murthy made changes -
        Target Version/s 0.23.1 [ 12318883 ]
        Priority Critical [ 2 ] Blocker [ 1 ]
        Mahadev konar made changes -
        Field Original Value New Value
        Priority Major [ 3 ] Critical [ 2 ]
        Siddharth Seth created issue -

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development