Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6703

Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes and specifically OPPORTUNISTIC containers.
      The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to provide hints via config to the MR framework as to the number of containers it would like to schedule as OPPORTUNISTIC.

      1. MAPREDUCE-6703.001.patch
        28 kB
        Arun Suresh
      2. MAPREDUCE-6703.002.patch
        27 kB
        Arun Suresh
      3. MAPREDUCE-6703.003.patch
        27 kB
        Arun Suresh

        Issue Links

          Activity

          Hide
          asuresh Arun Suresh added a comment -

          Uploading initial patch

          • Changes to allow the MRAppMaster to request a mix of OPPORTUNISTIC and GUARANTEED containers for it first attempt map tasks (reduce tasks and maps to be retried are always GUARANTEED)
          • User specifies number of OPPORTUNISTIC Containers via the mapreduce.job.num-opportunistic-maps-per-100 job conf : Default value is 0, which implies all maps will be guaranteed. A value of 100 means all maps will be requested as opportunistic. For any other value say x, the FIRST x maps requested by the AM will be opportunistic. If the total number of maps for the job < x, then ALL maps will be opportunistic
          • The patch also contains a test case TestMROpportunisticMaps that uses the MiniYARNCluster to run a simple external sorting MR job using the above flag.

          PS : This feature currently requires Distributed Scheduling (YARN-2885), NM Queuing (YARN-2883) and the AMRMProxyService (YARN-2884) to be turned on in the yarn cluster.

          Show
          asuresh Arun Suresh added a comment - Uploading initial patch Changes to allow the MRAppMaster to request a mix of OPPORTUNISTIC and GUARANTEED containers for it first attempt map tasks (reduce tasks and maps to be retried are always GUARANTEED) User specifies number of OPPORTUNISTIC Containers via the mapreduce.job.num-opportunistic-maps-per-100 job conf : Default value is 0, which implies all maps will be guaranteed. A value of 100 means all maps will be requested as opportunistic. For any other value say x , the FIRST x maps requested by the AM will be opportunistic. If the total number of maps for the job < x , then ALL maps will be opportunistic The patch also contains a test case TestMROpportunisticMaps that uses the MiniYARNCluster to run a simple external sorting MR job using the above flag. PS : This feature currently requires Distributed Scheduling ( YARN-2885 ), NM Queuing ( YARN-2883 ) and the AMRMProxyService ( YARN-2884 ) to be turned on in the yarn cluster.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 6m 36s trunk passed
          +1 compile 1m 36s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 49s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 1m 32s the patch passed
          +1 javac 1m 32s the patch passed
          -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 11 new + 640 unchanged - 18 fixed = 651 total (was 658)
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 56s the patch passed
          +1 javadoc 0m 41s the patch passed
          -1 unit 2m 0s hadoop-mapreduce-client-core in the patch failed.
          +1 unit 8m 24s hadoop-mapreduce-client-app in the patch passed.
          -1 unit 113m 46s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          146m 52s



          Reason Tests
          Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804646/MAPREDUCE-6703.001.patch
          JIRA Issue MAPREDUCE-6703
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cf46f41e5cf7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1597630
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 6m 36s trunk passed +1 compile 1m 36s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 49s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 1m 32s the patch passed +1 javac 1m 32s the patch passed -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 11 new + 640 unchanged - 18 fixed = 651 total (was 658) +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 26s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 2m 0s hadoop-mapreduce-client-core in the patch failed. +1 unit 8m 24s hadoop-mapreduce-client-app in the patch passed. -1 unit 113m 46s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 146m 52s Reason Tests Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804646/MAPREDUCE-6703.001.patch JIRA Issue MAPREDUCE-6703 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf46f41e5cf7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6512/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          looks like the locality is ignored for opportunistic containers, does YARN-2877 consider locality for opportunistic containers ?

          Show
          jianhe Jian He added a comment - looks like the locality is ignored for opportunistic containers, does YARN-2877 consider locality for opportunistic containers ?
          Hide
          asuresh Arun Suresh added a comment -

          Jian He, thanks for taking a look..

          So in the first version, we decided to ignore locality (and for that matter, node labels too), since the point of YARN-2877 is to improve allocation latencies at the cost of relaxing some scheduling constraints. The sweet-spot would be for Applications that have large number of really short lived tasks that do not care about locality.

          Reduce tasks are always guaranteed (since the cost of preempting a reduce task is quite high) and failed and retried map tasks are also always guaranteed..

          Show
          asuresh Arun Suresh added a comment - Jian He , thanks for taking a look.. So in the first version, we decided to ignore locality (and for that matter, node labels too), since the point of YARN-2877 is to improve allocation latencies at the cost of relaxing some scheduling constraints. The sweet-spot would be for Applications that have large number of really short lived tasks that do not care about locality. Reduce tasks are always guaranteed (since the cost of preempting a reduce task is quite high) and failed and retried map tasks are also always guaranteed..
          Hide
          jianhe Jian He added a comment -

          I see. depending on how locality sensitive the MR job is, this may not benefit as much. Wonder whether you have statistics to show how much this improves, or this is mainly for example purpose ?

          Show
          jianhe Jian He added a comment - I see. depending on how locality sensitive the MR job is, this may not benefit as much. Wonder whether you have statistics to show how much this improves, or this is mainly for example purpose ?
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Hi Jian He,
          This particular change in the MR AM is for people to start trying out OPPORTUNISTIC containers and distributed scheduling.
          However, you can find some of our initial experimental results on some internal workloads (consisting of both short and long tasks) in our paper here: http://research.microsoft.com/pubs/244469/mercury-atc2015.pdf.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Hi Jian He , This particular change in the MR AM is for people to start trying out OPPORTUNISTIC containers and distributed scheduling. However, you can find some of our initial experimental results on some internal workloads (consisting of both short and long tasks) in our paper here: http://research.microsoft.com/pubs/244469/mercury-atc2015.pdf .
          Hide
          jianhe Jian He added a comment -

          I see, just few minor comments:

          • could you add comments to the newly added config about what this config means ?
          • Here, we can just call addOpportunisticResourceRequest and so addContainerReq method does not need to be refactored.
            maps.put(event.getAttemptID(), request);

            addContainerReq(request, ExecutionType.OPPORTUNISTIC);
            
          Show
          jianhe Jian He added a comment - I see, just few minor comments: could you add comments to the newly added config about what this config means ? Here, we can just call addOpportunisticResourceRequest and so addContainerReq method does not need to be refactored. maps.put(event.getAttemptID(), request);
 addContainerReq(request, ExecutionType.OPPORTUNISTIC);
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 7m 41s trunk passed
          +1 compile 1m 48s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 0m 50s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 14s the patch passed
          +1 compile 1m 57s the patch passed
          +1 javac 1m 57s the patch passed
          -1 checkstyle 0m 35s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 11 new + 640 unchanged - 18 fixed = 651 total (was 658)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 0m 44s the patch passed
          +1 unit 2m 6s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 20s hadoop-mapreduce-client-app in the patch passed.
          -1 unit 119m 43s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          155m 14s



          Reason Tests
          Failed junit tests hadoop.mapred.TestMiniMRChildTask
          Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804646/MAPREDUCE-6703.001.patch
          JIRA Issue MAPREDUCE-6703
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 71f566c497aa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 42c22f7
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 7m 41s trunk passed +1 compile 1m 48s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 0m 50s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 1m 57s the patch passed +1 javac 1m 57s the patch passed -1 checkstyle 0m 35s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 11 new + 640 unchanged - 18 fixed = 651 total (was 658) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 2s the patch passed +1 javadoc 0m 44s the patch passed +1 unit 2m 6s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 20s hadoop-mapreduce-client-app in the patch passed. -1 unit 119m 43s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 155m 14s Reason Tests Failed junit tests hadoop.mapred.TestMiniMRChildTask Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804646/MAPREDUCE-6703.001.patch JIRA Issue MAPREDUCE-6703 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 71f566c497aa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 42c22f7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6516/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Attaching updated patch addressing Jian He's comments..

          Show
          asuresh Arun Suresh added a comment - Attaching updated patch addressing Jian He 's comments..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 17s trunk passed
          +1 compile 1m 35s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 16s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 0m 47s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 1m 33s the patch passed
          +1 javac 1m 33s the patch passed
          -1 checkstyle 0m 30s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 5 new + 642 unchanged - 18 fixed = 647 total (was 660)
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 1s the patch passed
          -1 javadoc 0m 19s hadoop-mapreduce-client-core in the patch failed.
          -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed.
          +1 unit 8m 9s hadoop-mapreduce-client-app in the patch passed.
          -1 unit 114m 6s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          146m 2s



          Reason Tests
          Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapred.TestMiniMRChildTask
            hadoop.mapred.TestLocalMRNotification



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805115/MAPREDUCE-6703.002.patch
          JIRA Issue MAPREDUCE-6703
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bf52daeb79d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 21d2b90
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 17s trunk passed +1 compile 1m 35s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 47s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 1m 33s the patch passed +1 javac 1m 33s the patch passed -1 checkstyle 0m 30s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 5 new + 642 unchanged - 18 fixed = 647 total (was 660) +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 1s the patch passed -1 javadoc 0m 19s hadoop-mapreduce-client-core in the patch failed. -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed. +1 unit 8m 9s hadoop-mapreduce-client-app in the patch passed. -1 unit 114m 6s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 146m 2s Reason Tests Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapred.TestMiniMRChildTask   hadoop.mapred.TestLocalMRNotification Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805115/MAPREDUCE-6703.002.patch JIRA Issue MAPREDUCE-6703 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf52daeb79d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21d2b90 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6517/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Attaching patch to fix the javadoc error and unused import checkstyle warning.
          Like I mentioned earlier, the tests run fine locally for me.

          Jian He, do let me know if you are fine with the latest patch, if so.. kindly do bless

          Show
          asuresh Arun Suresh added a comment - Attaching patch to fix the javadoc error and unused import checkstyle warning. Like I mentioned earlier, the tests run fine locally for me. Jian He , do let me know if you are fine with the latest patch, if so.. kindly do bless
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 6m 16s trunk passed
          +1 compile 1m 33s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 18s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 0m 49s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 1s the patch passed
          +1 compile 1m 30s the patch passed
          +1 javac 1m 30s the patch passed
          -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 4 new + 642 unchanged - 18 fixed = 646 total (was 660)
          +1 mvnsite 1m 10s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 0m 42s the patch passed
          -1 unit 1m 57s hadoop-mapreduce-client-core in the patch failed.
          +1 unit 8m 35s hadoop-mapreduce-client-app in the patch passed.
          -1 unit 115m 7s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          147m 32s



          Reason Tests
          Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805814/MAPREDUCE-6703.003.patch
          JIRA Issue MAPREDUCE-6703
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ae603bbb0b79 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b4078bd
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 16s trunk passed +1 compile 1m 33s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 18s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 49s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 1s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed -1 checkstyle 0m 31s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 4 new + 642 unchanged - 18 fixed = 646 total (was 660) +1 mvnsite 1m 10s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 42s the patch passed -1 unit 1m 57s hadoop-mapreduce-client-core in the patch failed. +1 unit 8m 35s hadoop-mapreduce-client-app in the patch passed. -1 unit 115m 7s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 147m 32s Reason Tests Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805814/MAPREDUCE-6703.003.patch JIRA Issue MAPREDUCE-6703 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ae603bbb0b79 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b4078bd Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6526/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          lgtm, I can commit later today if no comments from others.

          Show
          jianhe Jian He added a comment - lgtm, I can commit later today if no comments from others.
          Hide
          jianhe Jian He added a comment -

          Committed to trunk, branch-2, thanks Arun Suresh !

          Show
          jianhe Jian He added a comment - Committed to trunk, branch-2, thanks Arun Suresh !
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the review and the commit Jian He

          Show
          asuresh Arun Suresh added a comment - Thanks for the review and the commit Jian He
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9853 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9853/)
          MAPREDUCE-6703. Add flag to allow MapReduce AM to request for (jianhe: rev ae353ea96993ec664090c5d84f6675c29d9f0f5f)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMROpportunisticMaps.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9853 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9853/ ) MAPREDUCE-6703 . Add flag to allow MapReduce AM to request for (jianhe: rev ae353ea96993ec664090c5d84f6675c29d9f0f5f) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMROpportunisticMaps.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java

            People

            • Assignee:
              asuresh Arun Suresh
              Reporter:
              asuresh Arun Suresh
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development