Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5237

Fix missing log files issue in rolling log aggregation.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Steps to reproduce:
      1) enable RM recovery
      2) Run a sleep job
      3) restart RM
      4) kill the application
      We can not find that the logs for the first attempt

      1. YARN-5237.branch-2.8.patch
        12 kB
        Xuan Gong
      2. YARN-5237.4.patch
        12 kB
        Xuan Gong
      3. YARN-5237.3.patch
        10 kB
        Xuan Gong
      4. YARN-5237.2.patch
        10 kB
        Xuan Gong
      5. YARN-5237.1.patch
        5 kB
        Xuan Gong

        Activity

        Hide
        xgong Xuan Gong added a comment -

        Currently, we remove the finished container when we do the log aggregation which works fine if rolling log aggregation is disabled. Because the log aggregation only happens once.

        But, it would cause the issue if rolling log aggregation is enabled. Because, if we set includePattern/excludePattern, and we would not consider this during the application is still running. If the container finishes, and we remove the finished container, we would lose the chance to do the log aggregation (for includePattern/excludePattern) when the app finishes.

        this.pendingContainers.drainTo(pendingContainerInThisCycle);
        Set<ContainerId> finishedContainers =
            new HashSet<ContainerId>(pendingContainerInThisCycle);
        
        // This container is finished, and all its logs have been uploaded,
        // remove it from containerLogAggregators.
        if (finishedContainers.contains(container)) {
          containerLogAggregators.remove(container);
        }
        
        Show
        xgong Xuan Gong added a comment - Currently, we remove the finished container when we do the log aggregation which works fine if rolling log aggregation is disabled. Because the log aggregation only happens once. But, it would cause the issue if rolling log aggregation is enabled. Because, if we set includePattern/excludePattern, and we would not consider this during the application is still running. If the container finishes, and we remove the finished container, we would lose the chance to do the log aggregation (for includePattern/excludePattern) when the app finishes. this .pendingContainers.drainTo(pendingContainerInThisCycle); Set<ContainerId> finishedContainers = new HashSet<ContainerId>(pendingContainerInThisCycle); // This container is finished, and all its logs have been uploaded, // remove it from containerLogAggregators. if (finishedContainers.contains(container)) { containerLogAggregators.remove(container); }
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 31s trunk passed
        +1 compile 0m 26s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 41s trunk passed
        +1 javadoc 0m 17s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 23s the patch passed
        +1 javac 0m 23s the patch passed
        -1 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 170 unchanged - 1 fixed = 171 total (was 171)
        +1 mvnsite 0m 24s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 0m 46s the patch passed
        +1 javadoc 0m 14s the patch passed
        +1 unit 11m 56s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        24m 34s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809587/YARN-5237.1.patch
        JIRA Issue YARN-5237
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1c7de60022d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8a1dcce
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11984/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11984/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11984/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 31s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 41s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed -1 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 170 unchanged - 1 fixed = 171 total (was 171) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 11m 56s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 24m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809587/YARN-5237.1.patch JIRA Issue YARN-5237 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1c7de60022d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a1dcce Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11984/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11984/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11984/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        xgong Xuan Gong added a comment -

        Add more testcases

        Show
        xgong Xuan Gong added a comment - Add more testcases
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 23s trunk passed
        +1 compile 0m 26s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 43s trunk passed
        +1 javadoc 0m 17s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 23s the patch passed
        +1 javac 0m 23s the patch passed
        -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 6 new + 168 unchanged - 3 fixed = 174 total (was 171)
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 0m 46s the patch passed
        +1 javadoc 0m 15s the patch passed
        +1 unit 12m 10s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        24m 42s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809669/YARN-5237.2.patch
        JIRA Issue YARN-5237
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 08c25d901ec0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5279af7
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11995/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11995/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11995/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11995/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 23s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 6 new + 168 unchanged - 3 fixed = 174 total (was 171) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 12m 10s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 24m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809669/YARN-5237.2.patch JIRA Issue YARN-5237 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08c25d901ec0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5279af7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11995/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11995/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11995/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11995/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 36s trunk passed
        +1 compile 0m 30s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 49s trunk passed
        +1 javadoc 0m 19s trunk passed
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 2 new + 168 unchanged - 3 fixed = 170 total (was 171)
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 0m 54s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 13m 33s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        28m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809746/YARN-5237.3.patch
        JIRA Issue YARN-5237
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a226abfa6bc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 49b4064
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11997/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11997/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11997/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11997/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 36s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 2 new + 168 unchanged - 3 fixed = 170 total (was 171) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 54s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 13m 33s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 28m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809746/YARN-5237.3.patch JIRA Issue YARN-5237 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a226abfa6bc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 49b4064 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11997/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11997/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11997/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11997/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Hi Xuan Gong, thanks for reporting the issue and delivering the fix. I can understand the problem here, however, I am concerned that if we get rid of containerLogAggregators.remove(container); when containers get finished, it could end up with too many ContainerLogAggregators threads in case an app (long running service) allocate a lot of containers. Can we think some other way to address the problem? Something like: we can upload the container log to final location of HDFS in case of rolling log aggregation case (include pattern)?

        Show
        djp Junping Du added a comment - Hi Xuan Gong , thanks for reporting the issue and delivering the fix. I can understand the problem here, however, I am concerned that if we get rid of containerLogAggregators.remove(container); when containers get finished, it could end up with too many ContainerLogAggregators threads in case an app (long running service) allocate a lot of containers. Can we think some other way to address the problem? Something like: we can upload the container log to final location of HDFS in case of rolling log aggregation case (include pattern)?
        Hide
        xgong Xuan Gong added a comment -

        Can we think some other way to address the problem? Something like: we can upload the container log to final location of HDFS in case of rolling log aggregation case (include pattern)

        Yes, that makes sense. Uploaded a new patch to address your comment

        Show
        xgong Xuan Gong added a comment - Can we think some other way to address the problem? Something like: we can upload the container log to final location of HDFS in case of rolling log aggregation case (include pattern) Yes, that makes sense. Uploaded a new patch to address your comment
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 6m 15s trunk passed
        +1 compile 2m 1s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 1m 37s trunk passed
        +1 javadoc 0m 43s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 0m 46s the patch passed
        +1 compile 1m 58s the patch passed
        +1 javac 1m 58s the patch passed
        -1 checkstyle 0m 35s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 179 unchanged - 6 fixed = 181 total (was 185)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 22s the patch passed
        -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 44s the patch passed
        +1 unit 2m 14s hadoop-yarn-common in the patch passed.
        -1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        37m 19s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810067/YARN-5237.4.patch
        JIRA Issue YARN-5237
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 88012eafc9a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 709a814
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12006/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12006/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 6m 15s trunk passed +1 compile 2m 1s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 43s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 1m 58s the patch passed +1 javac 1m 58s the patch passed -1 checkstyle 0m 35s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 179 unchanged - 6 fixed = 181 total (was 185) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 22s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 44s the patch passed +1 unit 2m 14s hadoop-yarn-common in the patch passed. -1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 37m 19s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810067/YARN-5237.4.patch JIRA Issue YARN-5237 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 88012eafc9a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 709a814 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12006/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12006/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12006/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        xgong Xuan Gong added a comment -

        Testcase failure is not related.

        Show
        xgong Xuan Gong added a comment - Testcase failure is not related.
        Hide
        djp Junping Du added a comment -

        The checkstyle and whitespace issues reported by Jenkins is not valid.
        v4 patch LGTM. +1. Will commit patch shortly if no further comments from others.

        Show
        djp Junping Du added a comment - The checkstyle and whitespace issues reported by Jenkins is not valid. v4 patch LGTM. +1. Will commit patch shortly if no further comments from others.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 49s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 36s Maven dependency ordering for branch
        +1 mvninstall 7m 20s branch-2.8 passed
        +1 compile 1m 56s branch-2.8 passed with JDK v1.8.0_91
        +1 compile 2m 13s branch-2.8 passed with JDK v1.7.0_101
        +1 checkstyle 0m 35s branch-2.8 passed
        +1 mvnsite 1m 4s branch-2.8 passed
        +1 mvneclipse 0m 32s branch-2.8 passed
        +1 findbugs 2m 8s branch-2.8 passed
        +1 javadoc 0m 44s branch-2.8 passed with JDK v1.8.0_91
        +1 javadoc 0m 50s branch-2.8 passed with JDK v1.7.0_101
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 0m 50s the patch passed
        +1 compile 1m 52s the patch passed with JDK v1.8.0_91
        -1 javac 1m 52s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_91 with JDK v1.8.0_91 generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
        +1 compile 2m 14s the patch passed with JDK v1.7.0_101
        -1 javac 2m 14s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_101 with JDK v1.7.0_101 generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23)
        -1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 31 unchanged - 5 fixed = 32 total (was 36)
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 23s the patch passed
        +1 javadoc 0m 40s the patch passed with JDK v1.8.0_91
        +1 javadoc 0m 47s the patch passed with JDK v1.7.0_101
        +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_91.
        -1 unit 9m 2s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_91.
        +1 unit 2m 22s hadoop-yarn-common in the patch passed with JDK v1.7.0_101.
        -1 unit 9m 30s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        66m 17s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestNMProxy
        JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestNMProxy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c60792e
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810966/YARN-5237.branch-2.8.patch
        JIRA Issue YARN-5237
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0c796bc9f4c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 9128137
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_91.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_101.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_101.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12032/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12032/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 49s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 7m 20s branch-2.8 passed +1 compile 1m 56s branch-2.8 passed with JDK v1.8.0_91 +1 compile 2m 13s branch-2.8 passed with JDK v1.7.0_101 +1 checkstyle 0m 35s branch-2.8 passed +1 mvnsite 1m 4s branch-2.8 passed +1 mvneclipse 0m 32s branch-2.8 passed +1 findbugs 2m 8s branch-2.8 passed +1 javadoc 0m 44s branch-2.8 passed with JDK v1.8.0_91 +1 javadoc 0m 50s branch-2.8 passed with JDK v1.7.0_101 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 1m 52s the patch passed with JDK v1.8.0_91 -1 javac 1m 52s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_91 with JDK v1.8.0_91 generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) +1 compile 2m 14s the patch passed with JDK v1.7.0_101 -1 javac 2m 14s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_101 with JDK v1.7.0_101 generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) -1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 31 unchanged - 5 fixed = 32 total (was 36) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 23s the patch passed +1 javadoc 0m 40s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 47s the patch passed with JDK v1.7.0_101 +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_91. -1 unit 9m 2s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_91. +1 unit 2m 22s hadoop-yarn-common in the patch passed with JDK v1.7.0_101. -1 unit 9m 30s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 66m 17s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestNMProxy JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestNMProxy Subsystem Report/Notes Docker Image:yetus/hadoop:c60792e JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810966/YARN-5237.branch-2.8.patch JIRA Issue YARN-5237 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c796bc9f4c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 9128137 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_91.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_101.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/12032/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12032/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12032/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        +1 on branch-2.8 patch as well. Committing it in.

        Show
        djp Junping Du added a comment - +1 on branch-2.8 patch as well. Committing it in.
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Xuan Gong for patch contribution!

        Show
        djp Junping Du added a comment - I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Xuan Gong for patch contribution!

          People

          • Assignee:
            xgong Xuan Gong
            Reporter:
            sseth Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development