Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3404

Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: job submission, mrv2
    • Labels:
      None
    • Environment:

      Hadoop version is: Hadoop 0.23.0.1110031628
      10 node test cluster

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces.

      Description

      When forcing a mapper to take significantly longer than other map tasks, speculative map tasks are
      launched even if the mapreduce.job.maps.speculative.execution parameter is set to 'false'.

      Testcase: ran default WordCount job with spec execution set to false for both map and reduce but still saw a fifth mapper
      task launch, ran job as follows:

      hadoop --config <config> jar /tmp/testphw/wordcount.jar WordCount
      -Dmapreduce.job.maps.speculative.execution=false -Dmapreduce.job.reduces.speculative.execution=false
      /tmp/test_file_of_words* /tmp/file_of_words.out

      Input data was 4 text files >hdfs blocksize, with same word pattern plus one diff text line in each file, fourth
      file was 4 times as large as others:

      hadoop --config <config> fs -ls /tmp
      Found 5 items
      drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out
      rw-rr- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1
      rw-rr- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2
      rw-rr- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3
      rw-rr- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4

      Job launched 5 mappers despite spec exec set to false, output snippet:

      org.apache.hadoop.mapreduce.JobCounter
      NUM_FAILED_MAPS=1
      TOTAL_LAUNCHED_MAPS=5
      TOTAL_LAUNCHED_REDUCES=1
      RACK_LOCAL_MAPS=5
      SLOTS_MILLIS_MAPS=273540
      SLOTS_MILLIS_REDUCES=212876

      Reran same case as above only set both spec exec params to 'true', same results only this time the fifth task being
      launched is expected since spec exec = true.

      job run:

      hadoop --config <config> jar /tmp/testphw/wordcount.jar WordCount
      -Dmapreduce.job.maps.speculative.execution=true -Dmapreduce.job.reduces.speculative.execution=true
      /tmp/test_file_of_words* /tmp/file_of_words.out

      output snippet:

      org.apache.hadoop.mapreduce.JobCounter
      NUM_FAILED_MAPS=1
      TOTAL_LAUNCHED_MAPS=5
      TOTAL_LAUNCHED_REDUCES=1
      RACK_LOCAL_MAPS=5
      SLOTS_MILLIS_MAPS=279653
      SLOTS_MILLIS_REDUCES=211474

      1. MAPREDUCE-3404.1.txt
        11 kB
        Eric Payne
      2. MAPREDUCE-3404.2.txt
        14 kB
        Eric Payne

        Activity

        Hide
        Eric Payne added a comment -

        I have two corrections to make and three observation.

        • Corrections:
          • The property name that controls mapper speculative execution is mapreduce.map.speculative rather than
            mapreduce.job.maps.speculative.execution.
          • The property name that controls reduce speculative execution is mapreduce.reduce.speculative rather than
            mapreduce.job.reduces.speculative.execution.
        • Observations:
          • If the config values are set as follows in the mapred-site.xml file, then overriding of mapreduce.map.speculative
            works as expected on the command line:
              <property>
                <name>mapreduce.map.speculative</name>
                <value>true</value>
              </property>
            
              <property>
                <name>mapreduce.reduce.speculative</name>
                <value>false</value>
              </property>
            
            • That is, -Dmapreduce.map.speculative=false on the command line will override the value in the config file and turn
              off speculative map execution.
          • However, if mapreduce.reduce.speculative is set to true in the config file (as in the following example), it is not
            possible to turn off speculative map execution on the command line. For some reason, setting
            mapreduce.reduce.speculative to true in the config file overrides the command line value of mapreduce.map.speculative
              <property>
                <name>mapreduce.map.speculative</name>
                <value>true</value>
              </property>
            
              <property>
                <name>mapreduce.reduce.speculative</name>
                <value>true</value>
              </property>
            
          • Also, if the config file sets mapreduce.map.speculative to false but leaves mapreduce.reduce.speculative set to true, the speculative execution still happens for the mappers even though mapreduce.reduce.speculative is false. So, this is not just a command line override issue.
        Show
        Eric Payne added a comment - I have two corrections to make and three observation. Corrections: The property name that controls mapper speculative execution is mapreduce.map.speculative rather than mapreduce.job.maps.speculative.execution . The property name that controls reduce speculative execution is mapreduce.reduce.speculative rather than mapreduce.job.reduces.speculative.execution . Observations: If the config values are set as follows in the mapred-site.xml file, then overriding of mapreduce.map.speculative works as expected on the command line: <property> <name>mapreduce.map.speculative</name> <value>true</value> </property> <property> <name>mapreduce.reduce.speculative</name> <value>false</value> </property> That is, -Dmapreduce.map.speculative=false on the command line will override the value in the config file and turn off speculative map execution. However, if mapreduce.reduce.speculative is set to true in the config file (as in the following example), it is not possible to turn off speculative map execution on the command line. For some reason, setting mapreduce.reduce.speculative to true in the config file overrides the command line value of mapreduce.map.speculative <property> <name>mapreduce.map.speculative</name> <value>true</value> </property> <property> <name>mapreduce.reduce.speculative</name> <value>true</value> </property> Also, if the config file sets mapreduce.map.speculative to false but leaves mapreduce.reduce.speculative set to true , the speculative execution still happens for the mappers even though mapreduce.reduce.speculative is false . So, this is not just a command line override issue.
        Hide
        Eric Payne added a comment -

        In addition to the new unit test, I have successfully tested this patch on a 10-node cluster.

        Show
        Eric Payne added a comment - In addition to the new unit test, I have successfully tested this patch on a 10-node cluster.
        Hide
        Robert Joseph Evans added a comment -

        I have a couple of comments.

        1. In your test TestSpeculativeExecution.java line 65 (I think) and also later in the file, there is some code commented out. It should probably just be deleted.
        2. In the mapper code that checks for the first attempt of the first mapper, I would prefer to see it not do a string comparison as that value could change in the future. I would rather see it check the individual parts of the ID though the APIs it provides.
        3. Why are you calling Thread.currentThread() right before the call to sleep()? this should probably be deleted.
        4. It looks like there might be some tabs in the patch. Please replace them with spaces.
        5. The boolean expression
                    (   ( conf.getBoolean(MRJobConfig.MAP_SPECULATIVE, false)
                          && (tType == null || tType == TaskType.MAP) )
                     || ( conf.getBoolean(MRJobConfig.REDUCE_SPECULATIVE, false)
                        && (tType == null || tType == TaskType.REDUCE))           ) ) {
          

          is so complex that you needed comment to explain what it was doing. Also those conf values are not going to change from the time that the AM is launched. Could you please cache them. I think it would clean up the expression a lot.

        6. When will tType be null? and why do we want to pass it on to the Speculator when it is? I dug and it looks like it is null when the event type is JOB_CREATE or ATTEMPT_STATUS_UPDATE, but it would be good to document that in the javadocs for the Speculator interface (So that others can override that functionality in the future). It would also be cleaner imo to have the tType == null check be in a different place form the tType == Map or tType == REDUCE. But I really don't care so long as there is a comment explaining what is happening.
        Show
        Robert Joseph Evans added a comment - I have a couple of comments. In your test TestSpeculativeExecution.java line 65 (I think) and also later in the file, there is some code commented out. It should probably just be deleted. In the mapper code that checks for the first attempt of the first mapper, I would prefer to see it not do a string comparison as that value could change in the future. I would rather see it check the individual parts of the ID though the APIs it provides. Why are you calling Thread.currentThread() right before the call to sleep()? this should probably be deleted. It looks like there might be some tabs in the patch. Please replace them with spaces. The boolean expression ( ( conf.getBoolean(MRJobConfig.MAP_SPECULATIVE, false ) && (tType == null || tType == TaskType.MAP) ) || ( conf.getBoolean(MRJobConfig.REDUCE_SPECULATIVE, false ) && (tType == null || tType == TaskType.REDUCE)) ) ) { is so complex that you needed comment to explain what it was doing. Also those conf values are not going to change from the time that the AM is launched. Could you please cache them. I think it would clean up the expression a lot. When will tType be null? and why do we want to pass it on to the Speculator when it is? I dug and it looks like it is null when the event type is JOB_CREATE or ATTEMPT_STATUS_UPDATE, but it would be good to document that in the javadocs for the Speculator interface (So that others can override that functionality in the future). It would also be cleaner imo to have the tType == null check be in a different place form the tType == Map or tType == REDUCE. But I really don't care so long as there is a comment explaining what is happening.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507391/MAPREDUCE-3404.1.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1450//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1450//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507391/MAPREDUCE-3404.1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1450//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1450//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        @Eric,
        How do we make sure that if mapreduce.job.maps.speculative=false and mapreduce.job.reduce.speculative=true, the maps dont get speculated and the reduces gets speculated?

        Show
        Mahadev konar added a comment - @Eric, How do we make sure that if mapreduce.job.maps.speculative=false and mapreduce.job.reduce.speculative=true, the maps dont get speculated and the reduces gets speculated?
        Hide
        Arun C Murthy added a comment -

        Cancelling patch while Mahadev's comments are addressed.

        Show
        Arun C Murthy added a comment - Cancelling patch while Mahadev's comments are addressed.
        Hide
        Eric Payne added a comment -

        @Bobby,

        > 1. TestSpeculativeExecution.java: ... code commented out.
        Removed

        > 2. TestSpeculativeExecution.java: ... have mapper check the individual parts of the ID though the APIs...
        Done

        > 3. TestSpeculativeExecution.java: ... don't call Thread.currentThread() ...
        Removed.

        > 4. TestSpeculativeExecution.java: ... remove tabs ...
        Done.

        > 5. MRAppMaster.java: ... Simplify boolean expression ...
        Done.

        > 6. TestSpeculativeExecution.java: ... Override speculator so that it will cause speculation to happen ...
        Done.

        @Mahadev:
        > How do we make sure that if mapreduce.job.maps.speculative=false and mapreduce.job.reduce.speculative=true, the maps dont get speculated and the reduces gets speculated?
        I added a test case that covers speculation for reduces only.

        Show
        Eric Payne added a comment - @Bobby, > 1. TestSpeculativeExecution.java: ... code commented out. Removed > 2. TestSpeculativeExecution.java: ... have mapper check the individual parts of the ID though the APIs... Done > 3. TestSpeculativeExecution.java: ... don't call Thread.currentThread() ... Removed. > 4. TestSpeculativeExecution.java: ... remove tabs ... Done. > 5. MRAppMaster.java: ... Simplify boolean expression ... Done. > 6. TestSpeculativeExecution.java: ... Override speculator so that it will cause speculation to happen ... Done. @Mahadev: > How do we make sure that if mapreduce.job.maps.speculative=false and mapreduce.job.reduce.speculative=true, the maps dont get speculated and the reduces gets speculated? I added a test case that covers speculation for reduces only.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12510252/MAPREDUCE-3404.2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1595//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1595//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510252/MAPREDUCE-3404.2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1595//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1595//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        How do we make sure that if mapreduce.job.maps.speculative=false and mapreduce.job.reduce.speculative=true, the maps dont get speculated and the reduces gets speculated?

        The speculator handles map and reduce speculation separately. I just looked at the patch, and it achieves the above by not sending any map events to the speculative when map-speculation is disabled. The speculator doesn't seem to find any maps to speculates (as it doesn't know about any maps at all) and so only speculates reduces. Works (IMO) a convoluted way but can live with that.

        +1 for the patch. Pushing this in.

        Show
        Vinod Kumar Vavilapalli added a comment - How do we make sure that if mapreduce.job.maps.speculative=false and mapreduce.job.reduce.speculative=true, the maps dont get speculated and the reduces gets speculated? The speculator handles map and reduce speculation separately. I just looked at the patch, and it achieves the above by not sending any map events to the speculative when map-speculation is disabled. The speculator doesn't seem to find any maps to speculates (as it doesn't know about any maps at all) and so only speculates reduces. Works (IMO) a convoluted way but can live with that. +1 for the patch. Pushing this in.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        ..oh, and the tests look good too.

        Just committed this to trunk and branch-0.23. Thanks Eric!

        On a side note, not caused by this patch, it is not correct that we increment the num_failed_maps counter when the speculation kills a task. Instead we should have a num_killed_maps. Separate issue, will file a ticket.

        Show
        Vinod Kumar Vavilapalli added a comment - ..oh, and the tests look good too. Just committed this to trunk and branch-0.23. Thanks Eric! On a side note, not caused by this patch, it is not correct that we increment the num_failed_maps counter when the speculation kills a task. Instead we should have a num_killed_maps. Separate issue, will file a ticket.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #366 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/366/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.
        svn merge --ignore-ancestry -c 1231395 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #366 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/366/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. svn merge --ignore-ancestry -c 1231395 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1615 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1615/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1615 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1615/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1542 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1542/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1542 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1542/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #388 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/388/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.
        svn merge --ignore-ancestry -c 1231395 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #388 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/388/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. svn merge --ignore-ancestry -c 1231395 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1560 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1560/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1560 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1560/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #925 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #925 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #138 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.
        svn merge --ignore-ancestry -c 1231395 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #138 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. svn merge --ignore-ancestry -c 1231395 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #160 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.
        svn merge --ignore-ancestry -c 1231395 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #160 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. svn merge --ignore-ancestry -c 1231395 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231397 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #958 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/)
        MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #958 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/ ) MAPREDUCE-3404 . Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231395 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java

          People

          • Assignee:
            Eric Payne
            Reporter:
            patrick white
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development