Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added new plugin property yarn.nodemanager.disk-validator to allow the NodeManager to use an alternate class for checking whether a disk is good or not.

      Description

      It would be nice to have the option for a DiskChecker that has more sophisticated checking capabilities. In order to do this, we would first need DiskChecker to be pluggable.

      1. YARN-5137.001.patch
        13 kB
        Yufei Gu
      2. YARN-5137.002.patch
        8 kB
        Yufei Gu
      3. YARN-5137.003.patch
        8 kB
        Yufei Gu
      4. YARN-5137.004.patch
        10 kB
        Yufei Gu
      5. YARN-5137.005.patch
        14 kB
        Yufei Gu
      6. YARN-5137.006.patch
        10 kB
        Yufei Gu

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          I am closing this as a duplicate of YARN-4271 as that covers the pluggable health checker service for NM.

          Show
          subru Subru Krishnan added a comment - I am closing this as a duplicate of YARN-4271 as that covers the pluggable health checker service for NM.
          Hide
          rchiang Ray Chiang added a comment -

          Subru Krishnan NodeStatusHealthChecker already has the capability to change the script to the user's purpose, but DiskChecker is current not pluggable. While NodeStatusHealthChecker/LocalDirsHandlerService uses DiskChecker, there are other classes that call DiskChecker and could use the ability to plug in an alternate check. Or do you still see it as the same thing?

          Show
          rchiang Ray Chiang added a comment - Subru Krishnan NodeStatusHealthChecker already has the capability to change the script to the user's purpose, but DiskChecker is current not pluggable. While NodeStatusHealthChecker/LocalDirsHandlerService uses DiskChecker, there are other classes that call DiskChecker and could use the ability to plug in an alternate check. Or do you still see it as the same thing?
          Hide
          rmohan Raghav Mohan added a comment -

          Certain environments and systems require pluggable DiskChecker behavior in order to report out values to their environments, which check node health and other metrics. Making the LocalDirsHandlerService provides this capability.

          Show
          rmohan Raghav Mohan added a comment - Certain environments and systems require pluggable DiskChecker behavior in order to report out values to their environments, which check node health and other metrics. Making the LocalDirsHandlerService provides this capability.
          Hide
          rchiang Ray Chiang added a comment -

          I'm a little confused by the above statement.

          If we want more places to check the disk before writing, like ContainerLocalizer#download() but with an alternate DiskChecker, what would you suggest then?

          Show
          rchiang Ray Chiang added a comment - I'm a little confused by the above statement. If we want more places to check the disk before writing, like ContainerLocalizer#download() but with an alternate DiskChecker, what would you suggest then?
          Hide
          rchiang Ray Chiang added a comment -

          Reopening after discussion on YARN-4271.

          Show
          rchiang Ray Chiang added a comment - Reopening after discussion on YARN-4271 .
          Hide
          yufeigu Yufei Gu added a comment - - edited

          Patch 001 includes:

          • Add an new interface: DiskValidator
          • Add two classes to implement the interface: BasicDiskValidator and ReadWriteValidator, BasicDiskValidator invokes the existing DiskChecker. And ReadWriteValidator doesn't do anything right now, might make it functional in next JIRA.
          • Add one configuration in YarnConfiguration to indicate which DiskValidator Yarn want to use.
          • Migrate two callers of DiskChecker in YARN NodeManager to use new pluggable disk checker.
          Show
          yufeigu Yufei Gu added a comment - - edited Patch 001 includes: Add an new interface: DiskValidator Add two classes to implement the interface: BasicDiskValidator and ReadWriteValidator, BasicDiskValidator invokes the existing DiskChecker. And ReadWriteValidator doesn't do anything right now, might make it functional in next JIRA. Add one configuration in YarnConfiguration to indicate which DiskValidator Yarn want to use. Migrate two callers of DiskChecker in YARN NodeManager to use new pluggable disk checker.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 15s trunk passed
          +1 compile 6m 43s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 2m 31s trunk passed
          +1 mvneclipse 0m 47s trunk passed
          -1 findbugs 1m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 3s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 55s the patch passed
          +1 compile 6m 13s the patch passed
          +1 javac 6m 13s the patch passed
          -1 checkstyle 1m 26s root: The patch generated 9 new + 378 unchanged - 1 fixed = 387 total (was 379)
          +1 mvnsite 2m 14s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 25s the patch passed
          +1 javadoc 1m 52s the patch passed
          +1 unit 7m 6s hadoop-common in the patch passed.
          -1 unit 0m 25s hadoop-yarn-api in the patch failed.
          +1 unit 2m 8s hadoop-yarn-common in the patch passed.
          +1 unit 10m 42s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          65m 38s



          Reason Tests
          Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807578/YARN-5137.001.patch
          JIRA Issue YARN-5137
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 358e0cf4dfd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 97e2449
          Default Java 1.8.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11827/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11827/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 15s trunk passed +1 compile 6m 43s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 2m 31s trunk passed +1 mvneclipse 0m 47s trunk passed -1 findbugs 1m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 3s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 55s the patch passed +1 compile 6m 13s the patch passed +1 javac 6m 13s the patch passed -1 checkstyle 1m 26s root: The patch generated 9 new + 378 unchanged - 1 fixed = 387 total (was 379) +1 mvnsite 2m 14s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 25s the patch passed +1 javadoc 1m 52s the patch passed +1 unit 7m 6s hadoop-common in the patch passed. -1 unit 0m 25s hadoop-yarn-api in the patch failed. +1 unit 2m 8s hadoop-yarn-common in the patch passed. +1 unit 10m 42s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 65m 38s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807578/YARN-5137.001.patch JIRA Issue YARN-5137 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 358e0cf4dfd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 97e2449 Default Java 1.8.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11827/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11827/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11827/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Based on the offline discussion with Robert Kanter and Haibo Chen, I will split this JIRA, put the common part into HADOOP-13254 and leave the YARN side stuff in this JIRA.

          Show
          yufeigu Yufei Gu added a comment - Based on the offline discussion with Robert Kanter and Haibo Chen , I will split this JIRA, put the common part into HADOOP-13254 and leave the YARN side stuff in this JIRA.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s YARN-5137 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811478/YARN-5137.002.patch
          JIRA Issue YARN-5137
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12362/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-5137 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811478/YARN-5137.002.patch JIRA Issue YARN-5137 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12362/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 36s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 34s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 9s the patch passed
          +1 compile 2m 13s the patch passed
          +1 javac 2m 13s the patch passed
          +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 384 unchanged - 1 fixed = 384 total (was 385)
          +1 mvnsite 1m 17s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 53s the patch passed
          +1 javadoc 0m 55s the patch passed
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed.
          -1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          42m 29s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection
            hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks
            hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818916/YARN-5137.003.patch
          JIRA Issue YARN-5137
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 809fdabd193d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 557a245
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12443/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12443/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12443/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12443/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 36s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 34s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 2m 13s the patch passed +1 javac 2m 13s the patch passed +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 384 unchanged - 1 fixed = 384 total (was 385) +1 mvnsite 1m 17s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 53s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 15s hadoop-yarn-common in the patch passed. -1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 42m 29s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection   hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks   hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818916/YARN-5137.003.patch JIRA Issue YARN-5137 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 809fdabd193d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 557a245 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12443/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12443/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12443/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12443/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • In DirectoryCollection and ResourceLocalizationService, I recommend storing DiskValidatorFactory.getInstance(conf.get(YarnConfiguration.DISK_VALIDATOR)) as a member variable. The value is static, so we might as well cache it.
          • It might be worth adding a LOG.info() message at creation time to let the user know which type of DiskValidator was loaded.
          Show
          rchiang Ray Chiang added a comment - In DirectoryCollection and ResourceLocalizationService, I recommend storing DiskValidatorFactory.getInstance(conf.get(YarnConfiguration.DISK_VALIDATOR)) as a member variable. The value is static, so we might as well cache it. It might be worth adding a LOG.info() message at creation time to let the user know which type of DiskValidator was loaded.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Ray for the review. I uploaded patch 004 for all the comments.
          And the failed test cases are all unrelated.
          It is fine to not add new test cases, since the patch just wrap the previous disk checker.

          Show
          yufeigu Yufei Gu added a comment - Thanks Ray for the review. I uploaded patch 004 for all the comments. And the failed test cases are all unrelated. It is fine to not add new test cases, since the patch just wrap the previous disk checker.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 54s trunk passed
          +1 compile 2m 24s trunk passed
          +1 checkstyle 0m 43s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 2m 35s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 8s the patch passed
          +1 compile 2m 17s the patch passed
          +1 javac 2m 17s the patch passed
          +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 384 unchanged - 2 fixed = 384 total (was 386)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 59s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 2m 17s hadoop-yarn-common in the patch passed.
          -1 unit 13m 11s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          43m 12s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection
            hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819485/YARN-5137.004.patch
          JIRA Issue YARN-5137
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 733f45d8c9bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 43cf6b1
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12450/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12450/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12450/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12450/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 2m 24s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 2m 35s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 8s the patch passed +1 compile 2m 17s the patch passed +1 javac 2m 17s the patch passed +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 384 unchanged - 2 fixed = 384 total (was 386) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 59s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 17s hadoop-yarn-common in the patch passed. -1 unit 13m 11s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 43m 12s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection   hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819485/YARN-5137.004.patch JIRA Issue YARN-5137 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 733f45d8c9bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 43cf6b1 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12450/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12450/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12450/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12450/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 38s trunk passed
          +1 compile 2m 18s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 48s trunk passed
          +1 javadoc 1m 11s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 2m 14s the patch passed
          +1 javac 2m 14s the patch passed
          +1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 383 unchanged - 2 fixed = 383 total (was 385)
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 51s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 0m 22s hadoop-yarn-api in the patch passed.
          +1 unit 2m 14s hadoop-yarn-common in the patch passed.
          -1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          43m 2s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks
            hadoop.yarn.server.nodemanager.TestDirectoryCollection



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819496/YARN-5137.004.patch
          JIRA Issue YARN-5137
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 8c926df8d675 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 12aa184
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12451/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12451/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 48s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed +1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 383 unchanged - 2 fixed = 383 total (was 385) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 51s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 0m 22s hadoop-yarn-api in the patch passed. +1 unit 2m 14s hadoop-yarn-common in the patch passed. -1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 43m 2s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks   hadoop.yarn.server.nodemanager.TestDirectoryCollection Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819496/YARN-5137.004.patch JIRA Issue YARN-5137 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8c926df8d675 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 12aa184 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12451/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12451/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Another minor nit. In the constructor for ContainerLocalizer, it looks like all member variables except diskValidator have "this" in front of it.

          Show
          rchiang Ray Chiang added a comment - Another minor nit. In the constructor for ContainerLocalizer, it looks like all member variables except diskValidator have "this" in front of it.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks for the review, Ray Chiang. I've upload patch 005 for your comment.
          Besides, two test cases of TestLinuxContainerExecutorWithMocks did fail due to this patch. The reason they failed is they are defective (wrong assumption of what configuration files loaded). I fixed them in patch 005 as well.

          Show
          yufeigu Yufei Gu added a comment - Thanks for the review, Ray Chiang . I've upload patch 005 for your comment. Besides, two test cases of TestLinuxContainerExecutorWithMocks did fail due to this patch. The reason they failed is they are defective (wrong assumption of what configuration files loaded). I fixed them in patch 005 as well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 3m 55s Maven dependency ordering for branch
          +1 mvninstall 9m 38s trunk passed
          +1 compile 2m 44s trunk passed
          +1 checkstyle 0m 50s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 3m 11s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 14s the patch passed
          +1 compile 2m 51s the patch passed
          +1 javac 2m 51s the patch passed
          +1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 414 unchanged - 4 fixed = 414 total (was 418)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 28s the patch passed
          +1 javadoc 1m 6s the patch passed
          +1 unit 0m 29s hadoop-yarn-api in the patch passed.
          +1 unit 2m 32s hadoop-yarn-common in the patch passed.
          -1 unit 13m 17s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          52m 53s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection
            hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820038/YARN-5137.005.patch
          JIRA Issue YARN-5137
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux c5f1829c3956 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 703fdf8
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12490/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12490/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12490/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12490/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 3m 55s Maven dependency ordering for branch +1 mvninstall 9m 38s trunk passed +1 compile 2m 44s trunk passed +1 checkstyle 0m 50s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 3m 11s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 2m 51s the patch passed +1 javac 2m 51s the patch passed +1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 414 unchanged - 4 fixed = 414 total (was 418) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 6s the patch passed +1 unit 0m 29s hadoop-yarn-api in the patch passed. +1 unit 2m 32s hadoop-yarn-common in the patch passed. -1 unit 13m 17s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 52m 53s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection   hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820038/YARN-5137.005.patch JIRA Issue YARN-5137 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c5f1829c3956 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 703fdf8 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12490/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12490/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12490/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12490/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          The failed tests are unrelated.

          Show
          yufeigu Yufei Gu added a comment - The failed tests are unrelated.
          Hide
          rchiang Ray Chiang added a comment -

          Yeah, I think the fix to LinuxContainerExecutor and TestLinuxContainerExecutorWithMocks is sufficiently unrelated that it should go into its own JIRA.

          Varun Vasudev, it looks like YARN-4253 was the most recent modification to this code. Would you agree that this fix is what is intended with the interaction between containerSchedPriorityIsSet and LinuxContainerExecutor#setConf()? And should it go into a separate JIRA?

          Show
          rchiang Ray Chiang added a comment - Yeah, I think the fix to LinuxContainerExecutor and TestLinuxContainerExecutorWithMocks is sufficiently unrelated that it should go into its own JIRA. Varun Vasudev , it looks like YARN-4253 was the most recent modification to this code. Would you agree that this fix is what is intended with the interaction between containerSchedPriorityIsSet and LinuxContainerExecutor#setConf()? And should it go into a separate JIRA?
          Hide
          vvasudev Varun Vasudev added a comment -

          Can you explain why this was needed -

          -      Assert.assertEquals(result.size(), 18);
          +      Assert.assertEquals(result.size(), 19);
          

          The other change looks fine to me.

          Show
          vvasudev Varun Vasudev added a comment - Can you explain why this was needed - - Assert.assertEquals(result.size(), 18); + Assert.assertEquals(result.size(), 19); The other change looks fine to me.
          Hide
          yufeigu Yufei Gu added a comment - - edited

          Hi Varun Vasudev, thanks for the review.
          Since conf in class TestLinuxContainerExecutorWithMocks is changed to YarnConfiguration instead of Configuration, it adds one item(local dir) into result. So it is 19 instead of 18 now.
          I changed conf to an instance YarnConfiguration since the class TestLinuxContainerExecutorWithMocks has an unstable assumption that it won't load any yarn configuration files, e.g., yarn-default.xml and yarn-site.xml.

          Show
          yufeigu Yufei Gu added a comment - - edited Hi Varun Vasudev , thanks for the review. Since conf in class TestLinuxContainerExecutorWithMocks is changed to YarnConfiguration instead of Configuration , it adds one item(local dir) into result . So it is 19 instead of 18 now. I changed conf to an instance YarnConfiguration since the class TestLinuxContainerExecutorWithMocks has an unstable assumption that it won't load any yarn configuration files, e.g., yarn-default.xml and yarn-site.xml.
          Hide
          yufeigu Yufei Gu added a comment -

          Filed YARN-5444 for fix in LinuxContainerExecutor and TestLinuxContainerExecutorWithMocks, and this jira will depend on it.

          Show
          yufeigu Yufei Gu added a comment - Filed YARN-5444 for fix in LinuxContainerExecutor and TestLinuxContainerExecutorWithMocks, and this jira will depend on it.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks Varun Vasudev for your help on YARN-5444. I look forward to getting this fix unblocked.

          Show
          rchiang Ray Chiang added a comment - Thanks Varun Vasudev for your help on YARN-5444 . I look forward to getting this fix unblocked.
          Hide
          yufeigu Yufei Gu added a comment - - edited

          Patch 006 is the same to patch 005 except removing the bug fix code in YARN-5444.

          Show
          yufeigu Yufei Gu added a comment - - edited Patch 006 is the same to patch 005 except removing the bug fix code in YARN-5444 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 38s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 34s trunk passed
          +1 javadoc 1m 3s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 10s the patch passed
          +1 compile 2m 15s the patch passed
          +1 javac 2m 15s the patch passed
          +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 384 unchanged - 2 fixed = 384 total (was 386)
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 0m 22s hadoop-yarn-api in the patch passed.
          -1 unit 2m 14s hadoop-yarn-common in the patch failed.
          +1 unit 13m 24s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          42m 39s



          Reason Tests
          Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822670/YARN-5137.006.patch
          JIRA Issue YARN-5137
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux bfcc4693d2bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3cd386b
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12698/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12698/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 34s trunk passed +1 javadoc 1m 3s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 10s the patch passed +1 compile 2m 15s the patch passed +1 javac 2m 15s the patch passed +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 384 unchanged - 2 fixed = 384 total (was 386) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 52s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 0m 22s hadoop-yarn-api in the patch passed. -1 unit 2m 14s hadoop-yarn-common in the patch failed. +1 unit 13m 24s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 42m 39s Reason Tests Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822670/YARN-5137.006.patch JIRA Issue YARN-5137 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux bfcc4693d2bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3cd386b Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12698/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12698/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          LGTM. +1 (binding). Committing soon if there are no objections.

          Show
          rchiang Ray Chiang added a comment - LGTM. +1 (binding). Committing soon if there are no objections.
          Hide
          rchiang Ray Chiang added a comment -

          Committed to trunk and branch-2.

          Thanks Yufei Gu for the contribution and the spinoff JIRAs! Thanks Varun Vasudev, Robert Kanter, and Haibo Chen for your feedback and assistance!

          Show
          rchiang Ray Chiang added a comment - Committed to trunk and branch-2. Thanks Yufei Gu for the contribution and the spinoff JIRAs! Thanks Varun Vasudev , Robert Kanter , and Haibo Chen for your feedback and assistance!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10258 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10258/)
          YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via (rchiang: rev aea3e65749333b97d8c45280077f03d4ebddb5a8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10258 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10258/ ) YARN-5137 . Make DiskChecker pluggable in NodeManager. (Yufei Gu via (rchiang: rev aea3e65749333b97d8c45280077f03d4ebddb5a8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks a lot Ray Chiang for the review and commit. Thanks Robert Kanter, Haibo Chen and Varun Vasudev for the review.

          Show
          yufeigu Yufei Gu added a comment - Thanks a lot Ray Chiang for the review and commit. Thanks Robert Kanter , Haibo Chen and Varun Vasudev for the review.

            People

            • Assignee:
              yufeigu Yufei Gu
              Reporter:
              rchiang Ray Chiang
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development