Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6690

Limit the number of resources a single map reduce job can submit for localization

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Users will sometimes submit a large amount of resources to be localized as part of a single map reduce job. This can cause issues with YARN localization that destabilize the cluster and potentially impact other user jobs. These resources are specified via the files, libjars, archives and jobjar command line arguments or directly through the configuration (i.e. distributed cache api). The resources specified could be too large in multiple dimensions:

      1. Total size
      2. Number of files
      3. Size of an individual resource (i.e. a large fat jar)

      We would like to encourage good behavior on the client side by having the option of enforcing resource limits along the above dimensions.

      There should be a separate effort to enforce limits at the YARN layer on the server side, but this jira is only covering the map reduce layer on the client side. In practice, having these client side limits will get us a long way towards preventing these localization anti-patterns.

      1. MAPREDUCE-6690-trunk-v7.patch
        37 kB
        Chris Trezzo
      2. MAPREDUCE-6690-trunk-v6.patch
        34 kB
        Chris Trezzo
      3. MAPREDUCE-6690-trunk-v5.patch
        31 kB
        Chris Trezzo
      4. MAPREDUCE-6690-trunk-v4.patch
        32 kB
        Chris Trezzo
      5. MAPREDUCE-6690-trunk-v3.patch
        31 kB
        Chris Trezzo
      6. MAPREDUCE-6690-trunk-v2.patch
        20 kB
        Chris Trezzo
      7. MAPREDUCE-6690-trunk-v1.patch
        17 kB
        Chris Trezzo

        Issue Links

          Activity

          Hide
          ctrezzo Chris Trezzo added a comment -

          Attached is a v1 patch for this jira. A few notes:

          1. This patch enforces the Total size and Number of files dimensions.
          2. This patch does not support symlinks. They are broken in other places and it adds to the complexity of the patch if we want to support them (i.e. handling symlink cycles). If the community thinks symlink support is a blocker for this patch, please let me know.

          Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Attached is a v1 patch for this jira. A few notes: This patch enforces the Total size and Number of files dimensions. This patch does not support symlinks. They are broken in other places and it adds to the complexity of the patch if we want to support them (i.e. handling symlink cycles). If the community thinks symlink support is a blocker for this patch, please let me know. Thanks!
          Hide
          ctrezzo Chris Trezzo added a comment -

          Also note that this isn't perfect in that jobs could be of varying sizes and this is really only limiting the per-container amount of localization. At least, in the map reduce case, all of the containers localize the same amount, not considering differences in what is already in the YARN local cache on various node managers.

          Show
          ctrezzo Chris Trezzo added a comment - Also note that this isn't perfect in that jobs could be of varying sizes and this is really only limiting the per-container amount of localization. At least, in the map reduce case, all of the containers localize the same amount, not considering differences in what is already in the YARN local cache on various node managers.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 1m 30s trunk passed with JDK v1.8.0_91
          +1 compile 1m 42s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 30s trunk passed
          +1 javadoc 0m 32s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 39s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 47s the patch passed
          +1 compile 1m 32s the patch passed with JDK v1.8.0_91
          +1 javac 1m 32s the patch passed
          +1 compile 1m 41s the patch passed with JDK v1.7.0_95
          +1 javac 1m 41s the patch passed
          -1 checkstyle 0m 27s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 8 new + 537 unchanged - 0 fixed = 545 total (was 537)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 0s The patch has 1 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 53s the patch passed
          -1 javadoc 0m 22s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91.
          -1 javadoc 2m 18s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25)
          +1 javadoc 0m 36s the patch passed with JDK v1.7.0_95
          +1 unit 2m 9s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91.
          -1 unit 110m 55s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91.
          +1 unit 2m 31s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
          -1 unit 111m 15s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          252m 20s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.mapred.TestMiniMRChildTask
          JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802735/MAPREDUCE-6690-trunk-v1.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 7652e62211cc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1c5bbf6
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt
          javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 1m 30s trunk passed with JDK v1.8.0_91 +1 compile 1m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 30s trunk passed +1 javadoc 0m 32s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 39s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 47s the patch passed +1 compile 1m 32s the patch passed with JDK v1.8.0_91 +1 javac 1m 32s the patch passed +1 compile 1m 41s the patch passed with JDK v1.7.0_95 +1 javac 1m 41s the patch passed -1 checkstyle 0m 27s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 8 new + 537 unchanged - 0 fixed = 545 total (was 537) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 22s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 0s The patch has 1 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 53s the patch passed -1 javadoc 0m 22s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91. -1 javadoc 2m 18s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25) +1 javadoc 0m 36s the patch passed with JDK v1.7.0_95 +1 unit 2m 9s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. -1 unit 110m 55s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. +1 unit 2m 31s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. -1 unit 111m 15s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 252m 20s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.mapred.TestMiniMRChildTask JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802735/MAPREDUCE-6690-trunk-v1.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7652e62211cc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1c5bbf6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6485/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Attached is a v2 patch. Here are the updates:

          1. Fixed checkstyle and whitespace issues.
          2. Added check for limiting the size of an individual resource.
          3. Test failures from v1 were unrelated.
          Show
          ctrezzo Chris Trezzo added a comment - Attached is a v2 patch. Here are the updates: Fixed checkstyle and whitespace issues. Added check for limiting the size of an individual resource. Test failures from v1 were unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 37s trunk passed
          +1 compile 1m 29s trunk passed with JDK v1.8.0_91
          +1 compile 1m 42s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 30s trunk passed
          +1 javadoc 0m 32s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 38s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 46s the patch passed
          +1 compile 1m 25s the patch passed with JDK v1.8.0_91
          +1 javac 1m 25s the patch passed
          +1 compile 1m 37s the patch passed with JDK v1.7.0_95
          +1 javac 1m 37s the patch passed
          -1 checkstyle 0m 33s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 1 new + 577 unchanged - 0 fixed = 578 total (was 577)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 30s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 34s the patch passed with JDK v1.7.0_95
          -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91.
          -1 unit 109m 9s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91.
          +1 unit 2m 36s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
          -1 unit 130m 41s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          269m 24s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapred.TestMiniMRChildTask
          JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapred.TestMiniMRChildTask
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803314/MAPREDUCE-6690-trunk-v2.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 480670560c43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27242f2
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 37s trunk passed +1 compile 1m 29s trunk passed with JDK v1.8.0_91 +1 compile 1m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 30s trunk passed +1 javadoc 0m 32s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 38s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 1m 25s the patch passed with JDK v1.8.0_91 +1 javac 1m 25s the patch passed +1 compile 1m 37s the patch passed with JDK v1.7.0_95 +1 javac 1m 37s the patch passed -1 checkstyle 0m 33s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 1 new + 577 unchanged - 0 fixed = 578 total (was 577) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 30s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 34s the patch passed with JDK v1.7.0_95 -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91. -1 unit 109m 9s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. +1 unit 2m 36s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. -1 unit 130m 41s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 269m 24s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803314/MAPREDUCE-6690-trunk-v2.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 480670560c43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27242f2 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6492/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Chris! Initial comments:

          Is this intended to apply to all distributed cache items or only those that need to be uploaded during job submission? Some comments in the JIRA and the property descriptions imply it also should apply to items in the distributed cache that already reside in HDFS, but it doesn't look like the patch does that. The changes are to JobResourceUploader which AFAIK only gets involved on files that potentially need to be copied to the staging area before job submission. I'm not seeing how this affects items already in HDFS elsewhere before job submission (i.e.: items already in mapreduce.job.cache.*)

          Speaking of mapreduce.job.cache.*, it would be nice if the properties used that same prefix since it's related to the distributed cache. Also I'd personally prefer something like mapreduce.job.cache.limit.max-files, mapreduce.job.cache.limit.max-file-mb, and mapreduce.job.cache.limit.max-total-mb if it's supposed to apply to the entire distributed cache.

          The TotalNumberOfFilesAndSize API is verbose and error-prone – is there ever a valid reason to call incrementTotalSize without also calling incrementTotalNumberOfFiles and findMaxFileSize? Probably does the wrong thing if the client doesn't call them all for each file. IMHO there should just be two APIs, addFile(long filesize) and checkLimit(). Or maybe just one if it's OK to throw during addFile() directly.

          Suggestion: TotalNumberOfFilesAndSize might be easier to comprehend (and type) if named something like LimitsChecker. Also its constructor can just be passed a Configuration. Then it can hide all the confs and other implementation details related to the dist cache limits, and a predicate function like hasLimits() can be used to do the early-out checks. Or maybe we just pass it the files directly and it can decide internally whether to visit the paths or early-out.

          I think it would be very helpful if the file path was shown in the error message when something exceeds the single-file limit, otherwise the user has to manually track it down among all the files involved.

          Nit: Javadocs listing the parameters to a method but no description for any of those parameters isn't useful.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Chris! Initial comments: Is this intended to apply to all distributed cache items or only those that need to be uploaded during job submission? Some comments in the JIRA and the property descriptions imply it also should apply to items in the distributed cache that already reside in HDFS, but it doesn't look like the patch does that. The changes are to JobResourceUploader which AFAIK only gets involved on files that potentially need to be copied to the staging area before job submission. I'm not seeing how this affects items already in HDFS elsewhere before job submission (i.e.: items already in mapreduce.job.cache.*) Speaking of mapreduce.job.cache.*, it would be nice if the properties used that same prefix since it's related to the distributed cache. Also I'd personally prefer something like mapreduce.job.cache.limit.max-files, mapreduce.job.cache.limit.max-file-mb, and mapreduce.job.cache.limit.max-total-mb if it's supposed to apply to the entire distributed cache. The TotalNumberOfFilesAndSize API is verbose and error-prone – is there ever a valid reason to call incrementTotalSize without also calling incrementTotalNumberOfFiles and findMaxFileSize? Probably does the wrong thing if the client doesn't call them all for each file. IMHO there should just be two APIs, addFile(long filesize) and checkLimit(). Or maybe just one if it's OK to throw during addFile() directly. Suggestion: TotalNumberOfFilesAndSize might be easier to comprehend (and type) if named something like LimitsChecker. Also its constructor can just be passed a Configuration. Then it can hide all the confs and other implementation details related to the dist cache limits, and a predicate function like hasLimits() can be used to do the early-out checks. Or maybe we just pass it the files directly and it can decide internally whether to visit the paths or early-out. I think it would be very helpful if the file path was shown in the error message when something exceeds the single-file limit, otherwise the user has to manually track it down among all the files involved. Nit: Javadocs listing the parameters to a method but no description for any of those parameters isn't useful.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Thanks for the review Jason Lowe!

          Is this intended to apply to all distributed cache items or only those that need to be uploaded during job submission?

          Yes, it is intended to apply to all distributed cache items as well. Good catch! I will add in the DC items to the check. As a side note: the reasoning for including DC items is that even though the DC items are in an accessible place, they could still cause a significant amount of localization to the YARN local cache. The amount of localization is affected by the local cache size and the hit rate in the cache, but I chose to go with the most conservative approach.

          I will also address your other comments.

          Show
          ctrezzo Chris Trezzo added a comment - Thanks for the review Jason Lowe ! Is this intended to apply to all distributed cache items or only those that need to be uploaded during job submission? Yes, it is intended to apply to all distributed cache items as well. Good catch! I will add in the DC items to the check. As a side note: the reasoning for including DC items is that even though the DC items are in an accessible place, they could still cause a significant amount of localization to the YARN local cache. The amount of localization is affected by the local cache size and the hit rate in the cache, but I chose to go with the most conservative approach. I will also address your other comments.
          Hide
          jlowe Jason Lowe added a comment -

          I will add in the DC items to the check.

          This reminds me: should there be a corresponding YARN feature to reject applications that are asking for too much localization? Admins could then configure a cluster so it still rejects bad apps from frameworks that do not support this type of self-checking or from users who are overriding configs. This check in MapReduce would still be useful from the standpoint of avoiding large copies to HDFS for staging if we know it's not going to work anyway, but the YARN check could catch any type of application before the distributed cache bomb hits the cluster nodes when the bad app runs.

          Show
          jlowe Jason Lowe added a comment - I will add in the DC items to the check. This reminds me: should there be a corresponding YARN feature to reject applications that are asking for too much localization? Admins could then configure a cluster so it still rejects bad apps from frameworks that do not support this type of self-checking or from users who are overriding configs. This check in MapReduce would still be useful from the standpoint of avoiding large copies to HDFS for staging if we know it's not going to work anyway, but the YARN check could catch any type of application before the distributed cache bomb hits the cluster nodes when the bad app runs.
          Hide
          ctrezzo Chris Trezzo added a comment -

          should there be a corresponding YARN feature to reject applications that are asking for too much localization?

          Yes, totally agree. As per the description, I was thinking of creating a follow up jira for a more complete server-side YARN solution. I am thinking of something where we can leverage the container launch context and the node manager can be smart about not launching containers that will cause too much localization. I haven't thought too much about this yet, but I will definitely file the jira.

          Show
          ctrezzo Chris Trezzo added a comment - should there be a corresponding YARN feature to reject applications that are asking for too much localization? Yes, totally agree. As per the description, I was thinking of creating a follow up jira for a more complete server-side YARN solution. I am thinking of something where we can leverage the container launch context and the node manager can be smart about not launching containers that will cause too much localization. I haven't thought too much about this yet, but I will definitely file the jira.
          Hide
          ctrezzo Chris Trezzo added a comment -

          V3 attached.

          1. Addressed comments from Jason Lowe.
          2. Added more unit tests.
          Show
          ctrezzo Chris Trezzo added a comment - V3 attached. Addressed comments from Jason Lowe . Added more unit tests.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Note: I was getting some InvocationTargetException failures during TestMRJobs on my local machine. I am submitting the patch anyways to get a run.

          Show
          ctrezzo Chris Trezzo added a comment - Note: I was getting some InvocationTargetException failures during TestMRJobs on my local machine. I am submitting the patch anyways to get a run.
          Hide
          ctrezzo Chris Trezzo added a comment -

          I filled YARN-5192 to address the server-side YARN feature.

          Show
          ctrezzo Chris Trezzo added a comment - I filled YARN-5192 to address the server-side YARN feature.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 6m 14s trunk passed
          +1 compile 1m 30s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 32s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 40s the patch passed
          +1 compile 1m 28s the patch passed
          +1 javac 1m 28s the patch passed
          -1 checkstyle 0m 30s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new + 581 unchanged - 0 fixed = 582 total (was 581)
          +1 mvnsite 0m 46s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 16s the patch passed
          -1 javadoc 0m 19s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 2509 unchanged - 0 fixed = 2510 total (was 2509)
          +1 unit 1m 56s hadoop-mapreduce-client-core in the patch passed.
          -1 unit 124m 23s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          144m 49s



          Reason Tests
          Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapreduce.v2.TestMRJobs
            hadoop.mapred.TestMiniMRChildTask
            hadoop.mapreduce.v2.TestUberAM



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807595/MAPREDUCE-6690-trunk-v3.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e93aef851bf8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 16b1cc7
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 6m 14s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 32s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 40s the patch passed +1 compile 1m 28s the patch passed +1 javac 1m 28s the patch passed -1 checkstyle 0m 30s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new + 581 unchanged - 0 fixed = 582 total (was 581) +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 16s the patch passed -1 javadoc 0m 19s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 2509 unchanged - 0 fixed = 2510 total (was 2509) +1 unit 1m 56s hadoop-mapreduce-client-core in the patch passed. -1 unit 124m 23s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 144m 49s Reason Tests Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapreduce.v2.TestMRJobs   hadoop.mapred.TestMiniMRChildTask   hadoop.mapreduce.v2.TestUberAM Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807595/MAPREDUCE-6690-trunk-v3.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e93aef851bf8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 16b1cc7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6532/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Please suffer me a dumb question: assuming that YARN-5192 is implemented, why do we also need this JIRA? Doesn't having two settings to do the same thing from different ends make the system needlessly confusing?

          Show
          templedf Daniel Templeton added a comment - Please suffer me a dumb question: assuming that YARN-5192 is implemented, why do we also need this JIRA? Doesn't having two settings to do the same thing from different ends make the system needlessly confusing?
          Hide
          jlowe Jason Lowe added a comment -

          Implementing the check in MapReduce allows for fast-failure and more accurate/informative errors to the client. The check in MapReduce can prevent an unnecessary upload of one or more resources to the staging area in HDFS because the client knows the job is going to fail anyway. Also YARN-5192 will only be able to detect the error when a container starts to localize on a node that asks for a resource set that violate the limits. Since MapReduce localizes everything for all containers (including the AM) it will fail under YARN-5192 as soon as the AM tries to run on a node, but it might take a while for the AM to get scheduled. As for error reporting, if the violation comes from one or more files that were submitted locally then the paths via a YARN-5192 check will be for HDFS staging directories rather than the local path the client originally specified. The error also will not be reported to the job client submitting the job unless it hangs around to monitor the job after submission. With this check the job client will get the error directly when it tries to submit.

          If we don't care much about these differences then we can just go with the YARN-5192 implementation.

          Show
          jlowe Jason Lowe added a comment - Implementing the check in MapReduce allows for fast-failure and more accurate/informative errors to the client. The check in MapReduce can prevent an unnecessary upload of one or more resources to the staging area in HDFS because the client knows the job is going to fail anyway. Also YARN-5192 will only be able to detect the error when a container starts to localize on a node that asks for a resource set that violate the limits. Since MapReduce localizes everything for all containers (including the AM) it will fail under YARN-5192 as soon as the AM tries to run on a node, but it might take a while for the AM to get scheduled. As for error reporting, if the violation comes from one or more files that were submitted locally then the paths via a YARN-5192 check will be for HDFS staging directories rather than the local path the client originally specified. The error also will not be reported to the job client submitting the job unless it hangs around to monitor the job after submission. With this check the job client will get the error directly when it tries to submit. If we don't care much about these differences then we can just go with the YARN-5192 implementation.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the clarification, Jason Lowe. I assumed that YARN-5192 would implement the check as part of the submit call so that the client gets immediate feedback. The point that I forgot about, though, is that regardless the submit only happens after the resources have been uploaded to HDFS. Given that this check specifically targets wide loads, the cases where the server-side check would reject the submit are exactly the ones that will waste the most time with the upload.

          I now see the light. I would like to find a way, however, to try to keep the two settings in sync if possible. I've seen cases, such as the number of concurrent moves in the HDFS mover, where the limit is set on both the client and server sides, and it ends up confusing customers. What about having the RM offer up its resource limits through a call? The client could then query the RM's limits and apply those.

          Show
          templedf Daniel Templeton added a comment - Thanks for the clarification, Jason Lowe . I assumed that YARN-5192 would implement the check as part of the submit call so that the client gets immediate feedback. The point that I forgot about, though, is that regardless the submit only happens after the resources have been uploaded to HDFS. Given that this check specifically targets wide loads, the cases where the server-side check would reject the submit are exactly the ones that will waste the most time with the upload. I now see the light. I would like to find a way, however, to try to keep the two settings in sync if possible. I've seen cases, such as the number of concurrent moves in the HDFS mover, where the limit is set on both the client and server sides, and it ends up confusing customers. What about having the RM offer up its resource limits through a call? The client could then query the RM's limits and apply those.
          Hide
          jlowe Jason Lowe added a comment -

          I assumed that YARN-5192 would implement the check as part of the submit call so that the client gets immediate feedback.

          Note that YARN-5192 cannot do the check on application submit. An application submit only requires the resources necessary to get the ApplicationMaster localized. Subsequent containers for the application could have a completely different set of resources, and they won't be available in the application submission context for validation at submit time. MapReduce is an app framework that happens to localize all resources for all containers, but other application frameworks do not always do this.

          I would like to find a way, however, to try to keep the two settings in sync if possible.

          Agreed it would be annoying for admins to have to keep these in sync, assuming nobody would ever want to configure the YARN limit higher than the MapReduce limit.

          What about having the RM offer up its resource limits through a call?

          That would be one way to tackle it. There have been cases in the past where it would have been nice for clients to be able to query config settings via the central daemons (i.e.: namenode, resourcemanager, etc.) rather than assume the local settings in hdfs-site.xml or yarn-site.xml are the same as what the central daemon is using. That's a somewhat open-ended API change for YARN with backwards-compatibility concerns going forward, but maybe it's time we hammered out whether or not we're going to do it on a YARN JIRA and if not, what clients/users are supposed to do to better keep the client and the server in sync.

          Show
          jlowe Jason Lowe added a comment - I assumed that YARN-5192 would implement the check as part of the submit call so that the client gets immediate feedback. Note that YARN-5192 cannot do the check on application submit. An application submit only requires the resources necessary to get the ApplicationMaster localized. Subsequent containers for the application could have a completely different set of resources, and they won't be available in the application submission context for validation at submit time. MapReduce is an app framework that happens to localize all resources for all containers, but other application frameworks do not always do this. I would like to find a way, however, to try to keep the two settings in sync if possible. Agreed it would be annoying for admins to have to keep these in sync, assuming nobody would ever want to configure the YARN limit higher than the MapReduce limit. What about having the RM offer up its resource limits through a call? That would be one way to tackle it. There have been cases in the past where it would have been nice for clients to be able to query config settings via the central daemons (i.e.: namenode, resourcemanager, etc.) rather than assume the local settings in hdfs-site.xml or yarn-site.xml are the same as what the central daemon is using. That's a somewhat open-ended API change for YARN with backwards-compatibility concerns going forward, but maybe it's time we hammered out whether or not we're going to do it on a YARN JIRA and if not, what clients/users are supposed to do to better keep the client and the server in sync.
          Hide
          ctrezzo Chris Trezzo added a comment -

          V4 attached.

          1. Fixed checkstyle/javadoc.
          2. Fixed TestMRJobs failures (test only changes).
          Show
          ctrezzo Chris Trezzo added a comment - V4 attached. Fixed checkstyle/javadoc. Fixed TestMRJobs failures (test only changes).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 4s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809100/MAPREDUCE-6690-trunk-v4.patch
          JIRA Issue MAPREDUCE-6690
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6543/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 4s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809100/MAPREDUCE-6690-trunk-v4.patch JIRA Issue MAPREDUCE-6690 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6543/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 6m 20s trunk passed
          +1 compile 1m 30s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 1m 8s trunk passed
          +1 javadoc 0m 35s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 44s the patch passed
          +1 compile 1m 30s the patch passed
          +1 javac 1m 30s the patch passed
          +1 checkstyle 0m 31s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 20s the patch passed
          -1 javadoc 0m 19s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 2 new + 2508 unchanged - 1 fixed = 2510 total (was 2509)
          -1 unit 2m 1s hadoop-mapreduce-client-core in the patch failed.
          -1 unit 114m 22s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          135m 26s



          Reason Tests
          Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapred.TestMRCJCFileOutputCommitter



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809100/MAPREDUCE-6690-trunk-v4.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux f6e7eb5194a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1500a0a
          Default Java 1.8.0_91
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 6m 20s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 1m 8s trunk passed +1 javadoc 0m 35s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 44s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed +1 checkstyle 0m 31s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 20s the patch passed -1 javadoc 0m 19s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 2 new + 2508 unchanged - 1 fixed = 2510 total (was 2509) -1 unit 2m 1s hadoop-mapreduce-client-core in the patch failed. -1 unit 114m 22s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 135m 26s Reason Tests Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapred.TestMRCJCFileOutputCommitter Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809100/MAPREDUCE-6690-trunk-v4.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux f6e7eb5194a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1500a0a Default Java 1.8.0_91 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6544/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          V5 attached.

          1. Fixed javadoc.
          2. Test failures are unrelated.
          Show
          ctrezzo Chris Trezzo added a comment - V5 attached. Fixed javadoc. Test failures are unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 7m 47s trunk passed
          +1 compile 1m 52s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 0m 35s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 48s the patch passed
          +1 compile 1m 45s the patch passed
          +1 javac 1m 45s the patch passed
          +1 checkstyle 0m 30s the patch passed
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 25s the patch passed
          +1 javadoc 0m 29s the patch passed
          +1 unit 2m 1s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 122m 4s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          145m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809313/MAPREDUCE-6690-trunk-v5.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 22ced402a5f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9581fb7
          Default Java 1.8.0_91
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6546/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6546/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6546/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 7m 47s trunk passed +1 compile 1m 52s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 31s trunk passed +1 javadoc 0m 35s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 48s the patch passed +1 compile 1m 45s the patch passed +1 javac 1m 45s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 22s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 25s the patch passed +1 javadoc 0m 29s the patch passed +1 unit 2m 1s hadoop-mapreduce-client-core in the patch passed. +1 unit 122m 4s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 145m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809313/MAPREDUCE-6690-trunk-v5.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 22ced402a5f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9581fb7 Default Java 1.8.0_91 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6546/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6546/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6546/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          The whitespace errors were for lines that this patch did not touch. I am not sure why they appeared during the run. Jason Lowe the patch should be good as is, unless you have additional comments. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - The whitespace errors were for lines that this patch did not touch. I am not sure why they appeared during the run. Jason Lowe the patch should be good as is, unless you have additional comments. Thanks!
          Hide
          ctrezzo Chris Trezzo added a comment -

          V6 Attached. Re-based to trunk.

          Show
          ctrezzo Chris Trezzo added a comment - V6 Attached. Re-based to trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 8m 28s trunk passed
          +1 compile 2m 4s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 38s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 52s the patch passed
          +1 compile 1m 45s the patch passed
          +1 javac 1m 45s the patch passed
          +1 checkstyle 0m 31s the patch passed
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 42s the patch passed
          +1 javadoc 0m 34s the patch passed
          +1 unit 2m 15s hadoop-mapreduce-client-core in the patch passed.
          -1 unit 130m 39s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          156m 39s



          Reason Tests
          Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822196/MAPREDUCE-6690-trunk-v6.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux b7ef7637fc03 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 438a9f0
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 8m 28s trunk passed +1 compile 2m 4s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 38s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed +1 compile 1m 45s the patch passed +1 javac 1m 45s the patch passed +1 checkstyle 0m 31s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 34s the patch passed +1 unit 2m 15s hadoop-mapreduce-client-core in the patch passed. -1 unit 130m 39s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 156m 39s Reason Tests Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822196/MAPREDUCE-6690-trunk-v6.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b7ef7637fc03 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 438a9f0 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6659/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          TestMapReduceJobControl#testJobControlWithKillJob times out in trunk without this patch. The broken test is unrelated. I will file another jira to fix the test, but this patch should be ready for review. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - TestMapReduceJobControl#testJobControlWithKillJob times out in trunk without this patch. The broken test is unrelated. I will file another jira to fix the test, but this patch should be ready for review. Thanks!
          Hide
          ctrezzo Chris Trezzo added a comment -

          Filed jira to fix unrelated test failure: MAPREDUCE-6747

          Show
          ctrezzo Chris Trezzo added a comment - Filed jira to fix unrelated test failure: MAPREDUCE-6747
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch! Looks good overall with just a few nits:

          I think the code would be cleaner if we leveraged Configuration#getStringCollection to get the conf values rather than checking for null and splitting on comma directly. That method will return an empty collection if there are no values for the property, so then we can just remove some of the null checks and just loop over the items for each property. Some of the null checks would change to !isEmpty checks to avoid doing unnecessary mkdirs, etc., during upload methods, but they could be completely removed in the limit checking code.

          The totalConfigSize* variables are essentially loop-invariants, so they should be computed once in the Limits constructor rather than each addFile call.

          Both TestMRJobs and TestJobResourceUploader assume any IOException is OK if the job submission is supposed to fail. The unit tests should verify that the expected exception that failed the job submission was related to limits, otherwise we could be failing the job submission for the wrong reasons and the test would still pass. I'm thinking something along the lines of checking the exception message for limits-related wording, but maybe there's a cleaner way.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! Looks good overall with just a few nits: I think the code would be cleaner if we leveraged Configuration#getStringCollection to get the conf values rather than checking for null and splitting on comma directly. That method will return an empty collection if there are no values for the property, so then we can just remove some of the null checks and just loop over the items for each property. Some of the null checks would change to !isEmpty checks to avoid doing unnecessary mkdirs, etc., during upload methods, but they could be completely removed in the limit checking code. The totalConfigSize* variables are essentially loop-invariants, so they should be computed once in the Limits constructor rather than each addFile call. Both TestMRJobs and TestJobResourceUploader assume any IOException is OK if the job submission is supposed to fail. The unit tests should verify that the expected exception that failed the job submission was related to limits, otherwise we could be failing the job submission for the wrong reasons and the test would still pass. I'm thinking something along the lines of checking the exception message for limits-related wording, but maybe there's a cleaner way.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Thanks for the review Jason Lowe! Attached is a v7 patch. Here are the major changes:

          1. Changes to address your comments around getStringCollection, totalConfigSize* and ensuring tests failed in the intended way.
          2. Changes to make the usage of the word resource vs file consistent throughout the patch (i.e. a file is a type of resource).
          Show
          ctrezzo Chris Trezzo added a comment - Thanks for the review Jason Lowe ! Attached is a v7 patch. Here are the major changes: Changes to address your comments around getStringCollection, totalConfigSize* and ensuring tests failed in the intended way. Changes to make the usage of the word resource vs file consistent throughout the patch (i.e. a file is a type of resource).
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 1m 35s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 11s trunk passed
          +1 javadoc 0m 34s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 44s the patch passed
          +1 compile 1m 33s the patch passed
          +1 javac 1m 33s the patch passed
          +1 checkstyle 0m 31s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 19s the patch passed
          +1 javadoc 0m 30s the patch passed
          +1 unit 2m 11s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 118m 49s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          140m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824023/MAPREDUCE-6690-trunk-v7.patch
          JIRA Issue MAPREDUCE-6690
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2bad52d01f13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27a6e09
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6673/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6673/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 1m 35s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 11s trunk passed +1 javadoc 0m 34s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 44s the patch passed +1 compile 1m 33s the patch passed +1 javac 1m 33s the patch passed +1 checkstyle 0m 31s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 19s the patch passed +1 javadoc 0m 30s the patch passed +1 unit 2m 11s hadoop-mapreduce-client-core in the patch passed. +1 unit 118m 49s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 140m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824023/MAPREDUCE-6690-trunk-v7.patch JIRA Issue MAPREDUCE-6690 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2bad52d01f13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27a6e09 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6673/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6673/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Chris! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks, Chris! I committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10291 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10291/)
          MAPREDUCE-6690. Limit the number of resources a single map reduce job (jlowe: rev f80a7298325a4626638ee24467e2012442e480d4)

          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • (add) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploader.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10291 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10291/ ) MAPREDUCE-6690 . Limit the number of resources a single map reduce job (jlowe: rev f80a7298325a4626638ee24467e2012442e480d4) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java (add) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploader.java
          Hide
          ctrezzo Chris Trezzo added a comment -

          Thanks Jason Lowe for the review and commit!

          Show
          ctrezzo Chris Trezzo added a comment - Thanks Jason Lowe for the review and commit!

            People

            • Assignee:
              ctrezzo Chris Trezzo
              Reporter:
              ctrezzo Chris Trezzo
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development