Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3490

RMContainerAllocator counts failed maps towards Reduce ramp up

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed MapReduce AM to count failed maps also towards Reduce ramp up.

      Description

      The RMContainerAllocator does not differentiate between failed and successful maps while calculating whether reduce tasks are ready to launch. Failed tasks are also counted towards total completed tasks.
      Example. 4 failed maps, 10 total maps. Map%complete = 4/14 * 100 instead of being 0.

      1. MAPREDUCE-3490.patch
        7 kB
        Arun C Murthy
      2. MAPREDUCE-3490.patch
        11 kB
        Arun C Murthy
      3. MAPREDUCE-3490.patch
        12 kB
        Arun C Murthy
      4. MAPREDUCE-3490.patch
        15 kB
        Arun C Murthy
      5. MR-3490-alternate.patch
        3 kB
        Sharad Agarwal
      6. MR-3490-alternate1.patch
        5 kB
        Sharad Agarwal
      7. MAPREDUCE-3490.patch
        19 kB
        Arun C Murthy
      8. MAPREDUCE-3490.patch
        19 kB
        Arun C Murthy
      9. MAPREDUCE-3490.patch
        13 kB
        Arun C Murthy

        Activity

        Hide
        Arun C Murthy added a comment -

        I'll take this one, seems very important to fix.

        Show
        Arun C Murthy added a comment - I'll take this one, seems very important to fix.
        Hide
        Arun C Murthy added a comment -

        Straight-fwd patch, testing now. I'll update after with unit tests.

        Show
        Arun C Murthy added a comment - Straight-fwd patch, testing now. I'll update after with unit tests.
        Hide
        Arun C Murthy added a comment -

        Updated with a unit test, tested on a secure cluster.

        Show
        Arun C Murthy added a comment - Updated with a unit test, tested on a secure cluster.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12508327/MAPREDUCE-3490.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 2310 javac compiler warnings (more than the trunk's current 2308 warnings).

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.mapreduce.v2.app.TestFail
        org.apache.hadoop.mapreduce.v2.app.TestKill
        org.apache.hadoop.mapreduce.v2.app.TestRecovery
        org.apache.hadoop.mapreduce.v2.app.TestFetchFailure

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1491//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1491//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508327/MAPREDUCE-3490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2310 javac compiler warnings (more than the trunk's current 2308 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.app.TestFail org.apache.hadoop.mapreduce.v2.app.TestKill org.apache.hadoop.mapreduce.v2.app.TestRecovery org.apache.hadoop.mapreduce.v2.app.TestFetchFailure +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1491//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1491//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12508329/MAPREDUCE-3490.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 2310 javac compiler warnings (more than the trunk's current 2308 warnings).

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.mapreduce.v2.app.TestKill
        org.apache.hadoop.mapreduce.v2.app.TestRecovery
        org.apache.hadoop.mapreduce.v2.app.TestFail
        org.apache.hadoop.mapreduce.v2.app.TestFetchFailure

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1492//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1492//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508329/MAPREDUCE-3490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2310 javac compiler warnings (more than the trunk's current 2308 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.app.TestKill org.apache.hadoop.mapreduce.v2.app.TestRecovery org.apache.hadoop.mapreduce.v2.app.TestFail org.apache.hadoop.mapreduce.v2.app.TestFetchFailure +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1492//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1492//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        Had to fix MRApp to ensure this patch didn't break other tests.

        Show
        Arun C Murthy added a comment - Had to fix MRApp to ensure this patch didn't break other tests.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12508338/MAPREDUCE-3490.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 2310 javac compiler warnings (more than the trunk's current 2308 warnings).

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1493//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1493//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508338/MAPREDUCE-3490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2310 javac compiler warnings (more than the trunk's current 2308 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1493//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1493//console This message is automatically generated.
        Hide
        Sharad Agarwal added a comment -

        Hi Arun - just had a brief look at the patch. seems like you don't need new Container Allocator types. RMContainerAllocator is already getting CONTAINER_FAILED event. completedMaps includes succeeded and failed.

        Instead of incorporating completedMaps in the calculation, it can take succeededMaps.

        succeededMaps = completedMaps - failedMaps

        Show
        Sharad Agarwal added a comment - Hi Arun - just had a brief look at the patch. seems like you don't need new Container Allocator types. RMContainerAllocator is already getting CONTAINER_FAILED event. completedMaps includes succeeded and failed. Instead of incorporating completedMaps in the calculation, it can take succeededMaps. succeededMaps = completedMaps - failedMaps
        Hide
        Sharad Agarwal added a comment -

        Here is the quick patch for illustration.

        Show
        Sharad Agarwal added a comment - Here is the quick patch for illustration.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12508387/MR-3490-alternate.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1494//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1494//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508387/MR-3490-alternate.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1494//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1494//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        Sharad, eventually I think we need to stop tracking this in RMContainerAllocator and rather rely on Job. For now, my patch seems the closest approximation to that (being conservative). Does that make sense? I'll file a separate jira to stop tracking these in RMContainerAllocator.

        Show
        Arun C Murthy added a comment - Sharad, eventually I think we need to stop tracking this in RMContainerAllocator and rather rely on Job. For now, my patch seems the closest approximation to that (being conservative). Does that make sense? I'll file a separate jira to stop tracking these in RMContainerAllocator.
        Hide
        Sharad Agarwal added a comment -

        I think we need to stop tracking this in RMContainerAllocator and rather rely on Job. For now, my patch seems the closest approximation to that (being conservative).

        Doing it in Job or in RMContainerAllocator is a separate discussion. I don't think this patch deal with anything like that. It adds two new events for RMContainerAllocator itself.
        I am proposing that we don't need these extra events because this information (failed attempts info) is already available in RMContainerAllocator.

        Show
        Sharad Agarwal added a comment - I think we need to stop tracking this in RMContainerAllocator and rather rely on Job. For now, my patch seems the closest approximation to that (being conservative). Doing it in Job or in RMContainerAllocator is a separate discussion. I don't think this patch deal with anything like that. It adds two new events for RMContainerAllocator itself. I am proposing that we don't need these extra events because this information (failed attempts info) is already available in RMContainerAllocator.
        Hide
        Arun C Murthy added a comment -

        Sharad, sorry, I should have explained better.

        The problem computing success/failed tasks in the RMContainerAllocator is that they only know Container status (at best exit code) and not whether the task completed successfully or not. Thus, we'd have to duplicate logic to compute tasks' status from container statuses there, something I'd like to avoid.

        Thus, my proposal to rely on JobImpl (either via events or just a direct function call). Makes sense?

        Thoughts?

        Show
        Arun C Murthy added a comment - Sharad, sorry, I should have explained better. The problem computing success/failed tasks in the RMContainerAllocator is that they only know Container status (at best exit code) and not whether the task completed successfully or not. Thus, we'd have to duplicate logic to compute tasks' status from container statuses there, something I'd like to avoid. Thus, my proposal to rely on JobImpl (either via events or just a direct function call). Makes sense? Thoughts?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Yes, we need to look at task finish states instead of attempts. +1 for that in general.

        We can separate the discussion of new events vs looking up in job.

        We can rename the new events in the patch to TASK_FINISHED, and MAP_TASK_RETROACTIVELY_FAILED or something like that.

        We also need to send the event irrespective of task success/fail state. In cases where task failures are ignorable, we don't want to block reduces waiting to launch new attempts for those failed tasks.

        Show
        Vinod Kumar Vavilapalli added a comment - Yes, we need to look at task finish states instead of attempts. +1 for that in general. We can separate the discussion of new events vs looking up in job. We can rename the new events in the patch to TASK_FINISHED, and MAP_TASK_RETROACTIVELY_FAILED or something like that. We also need to send the event irrespective of task success/fail state. In cases where task failures are ignorable, we don't want to block reduces waiting to launch new attempts for those failed tasks.
        Hide
        Sharad Agarwal added a comment -

        currently all bookkeeping and calculations in RMContainerAllocator are based on attempts not on tasks. there is one to one between attempt and container.

        completedMapPercent is currently completed map attempts percentage. I looked in more detail. we can simply change this to reflect completed map tasks percentage. All information is readily available there in Job. So we need not maintain these counts in RMContainerAllocator. Arun also mentioned this. I am attaching a patch which drastically simplify this, without the need to add new events. Also I have removed the completedMaps and completedReduces counts in RMContainerAllocator.
        Arun/Vinod - see if this make sense ?

        Show
        Sharad Agarwal added a comment - currently all bookkeeping and calculations in RMContainerAllocator are based on attempts not on tasks. there is one to one between attempt and container. completedMapPercent is currently completed map attempts percentage. I looked in more detail. we can simply change this to reflect completed map tasks percentage. All information is readily available there in Job. So we need not maintain these counts in RMContainerAllocator. Arun also mentioned this. I am attaching a patch which drastically simplify this, without the need to add new events. Also I have removed the completedMaps and completedReduces counts in RMContainerAllocator. Arun/Vinod - see if this make sense ?
        Hide
        Sharad Agarwal added a comment -

        Note: I am on vacation rest of the week. please expect slow or no response till then.

        Show
        Sharad Agarwal added a comment - Note: I am on vacation rest of the week. please expect slow or no response till then.
        Hide
        Arun C Murthy added a comment -

        Thanks Sharad. I've updated your patch with unit tests. All of the re-factoring of RMContainerAllocator.scheduleReduces was to aid unit tests.

        Show
        Arun C Murthy added a comment - Thanks Sharad. I've updated your patch with unit tests. All of the re-factoring of RMContainerAllocator.scheduleReduces was to aid unit tests.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509171/MAPREDUCE-3490.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1523//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1523//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509171/MAPREDUCE-3490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1523//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1523//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        Minor change. I'll add the log of 'successful maps/reduces' to RMContainerAllocator.getStats.

        Show
        Arun C Murthy added a comment - Minor change. I'll add the log of 'successful maps/reduces' to RMContainerAllocator.getStats.
        Hide
        Arun C Murthy added a comment -

        Added the logs back to aid diagnosis in future.

        Show
        Arun C Murthy added a comment - Added the logs back to aid diagnosis in future.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509178/MAPREDUCE-3490.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1524//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1524//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509178/MAPREDUCE-3490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1524//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1524//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I am attaching a patch which drastically simplify this, without the need to add new events. Also I have removed the completedMaps and completedReduces counts in RMContainerAllocator. Arun/Vinod - see if this make sense ?

        Thanks for the explanation, Sharad. Makes sense. Obviously we missed the big picture here At any rate, this code definitely needs some cleanup, way too complicated for my simple mind to track all of it

        Thanks for the update, Arun. Looking at the patch now.

        Show
        Vinod Kumar Vavilapalli added a comment - I am attaching a patch which drastically simplify this, without the need to add new events. Also I have removed the completedMaps and completedReduces counts in RMContainerAllocator. Arun/Vinod - see if this make sense ? Thanks for the explanation, Sharad. Makes sense. Obviously we missed the big picture here At any rate, this code definitely needs some cleanup, way too complicated for my simple mind to track all of it Thanks for the update, Arun. Looking at the patch now.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Patch looks better now. Except for the fact that we should continue to track completed tasks and not just successfulTasks to accommodate jobs for which task-failures are permissible.

        Show
        Vinod Kumar Vavilapalli added a comment - Patch looks better now. Except for the fact that we should continue to track completed tasks and not just successfulTasks to accommodate jobs for which task-failures are permissible.
        Hide
        Arun C Murthy added a comment -

        Thanks Vinod, missed that.

        I've updated the patch to use Job.completedMaps rather than Job.successfulMaps.

        Show
        Arun C Murthy added a comment - Thanks Vinod, missed that. I've updated the patch to use Job.completedMaps rather than Job.successfulMaps.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509362/MAPREDUCE-3490.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1532//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1532//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509362/MAPREDUCE-3490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1532//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1532//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        This looks good now. +1.

        Show
        Vinod Kumar Vavilapalli added a comment - This looks good now. +1.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I just committed this to trunk and branch-0.23. Thanks Sharad and Arun!

        Show
        Vinod Kumar Vavilapalli added a comment - I just committed this to trunk and branch-0.23. Thanks Sharad and Arun!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1564 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1564/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1564 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1564/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1492 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1492/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1492 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1492/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #324 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/324/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.
        svn merge -c 1227226 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #324 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/324/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. svn merge -c 1227226 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #335 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/335/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.
        svn merge -c 1227226 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #335 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/335/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. svn merge -c 1227226 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #346 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/346/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.
        svn merge -c 1227226 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #346 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/346/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. svn merge -c 1227226 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1513 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1513/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1513 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1513/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #916 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/916/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #916 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/916/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #129 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/129/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.
        svn merge -c 1227226 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #129 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/129/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. svn merge -c 1227226 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #151 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/151/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.
        svn merge -c 1227226 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #151 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/151/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. svn merge -c 1227226 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227227 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #949 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/949/)
        MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #949 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/949/ ) MAPREDUCE-3490 . Fixed MapReduce AM to count failed maps also towards Reduce ramp up. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227226 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development