Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-beta1
    • None
    • None
    • Reviewed

    Description

      Currently, TFile log reader is used to read aggregated log in YARN. We need to add an abstract layer, and pick up the correct log reader based on the configuration.

      Attachments

        1. YARN-6877-branch-2.001.patch
          82 kB
          Xuan Gong
        2. YARN-6877-branch-2.002.patch
          115 kB
          Xuan Gong
        3. YARN-6877-trunk.001.patch
          84 kB
          Xuan Gong
        4. YARN-6877-trunk.002.patch
          89 kB
          Xuan Gong
        5. YARN-6877-trunk.003.patch
          115 kB
          Xuan Gong
        6. YARN-6877-trunk.004.patch
          115 kB
          Xuan Gong
        7. YARN-6877-trunk.005.patch
          119 kB
          Xuan Gong
        8. YARN-6877-trunk.006.patch
          116 kB
          Xuan Gong
        9. YARN-6877-trunk.007.patch
          116 kB
          Xuan Gong
        10. YARN-6877-trunk.008.patch
          116 kB
          Xuan Gong

        Issue Links

          Activity

            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 17s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for branch
            +1 mvninstall 13m 6s trunk passed
            +1 compile 9m 10s trunk passed
            +1 checkstyle 0m 56s trunk passed
            +1 mvnsite 2m 14s trunk passed
            -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 57s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 33s the patch passed
            +1 compile 5m 41s the patch passed
            +1 javac 5m 41s the patch passed
            -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 19 new + 137 unchanged - 10 fixed = 156 total (was 147)
            +1 mvnsite 2m 11s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            -1 findbugs 1m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
            -1 javadoc 0m 42s hadoop-yarn-common in the patch failed.
                  Other Tests
            -1 unit 2m 42s hadoop-yarn-common in the patch failed.
            +1 unit 14m 26s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 10s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            +1 unit 20m 54s hadoop-yarn-client in the patch passed.
            +1 asflicense 0m 29s The patch does not generate ASF License warnings.
            98m 3s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
              org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationTFileController$TFileAggregatedLogsBlock.render(HtmlBlock$Block) might ignore java.lang.Exception At LogAggregationTFileController.java:At LogAggregationTFileController.java:[line 440]
              Exception is caught when Exception is not thrown in org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationTFileController$TFileAggregatedLogsBlock.render(HtmlBlock$Block) At LogAggregationTFileController.java:is not thrown in org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationTFileController$TFileAggregatedLogsBlock.render(HtmlBlock$Block) At LogAggregationTFileController.java:[line 440]
            Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogDeletionService
              hadoop.yarn.logaggregation.TestAggregatedLogsBlock



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883609/YARN-6877-trunk.002.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux a9d5bd9f11ab 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / c2cb7ea
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17125/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17125/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 13m 6s trunk passed +1 compile 9m 10s trunk passed +1 checkstyle 0m 56s trunk passed +1 mvnsite 2m 14s trunk passed -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 57s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 5m 41s the patch passed +1 javac 5m 41s the patch passed -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 19 new + 137 unchanged - 10 fixed = 156 total (was 147) +1 mvnsite 2m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) -1 javadoc 0m 42s hadoop-yarn-common in the patch failed.       Other Tests -1 unit 2m 42s hadoop-yarn-common in the patch failed. +1 unit 14m 26s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 10s hadoop-yarn-server-applicationhistoryservice in the patch passed. +1 unit 20m 54s hadoop-yarn-client in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 98m 3s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common   org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationTFileController$TFileAggregatedLogsBlock.render(HtmlBlock$Block) might ignore java.lang.Exception At LogAggregationTFileController.java:At LogAggregationTFileController.java: [line 440]   Exception is caught when Exception is not thrown in org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationTFileController$TFileAggregatedLogsBlock.render(HtmlBlock$Block) At LogAggregationTFileController.java:is not thrown in org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationTFileController$TFileAggregatedLogsBlock.render(HtmlBlock$Block) At LogAggregationTFileController.java: [line 440] Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogDeletionService   hadoop.yarn.logaggregation.TestAggregatedLogsBlock Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883609/YARN-6877-trunk.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a9d5bd9f11ab 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c2cb7ea Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17125/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17125/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17125/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            xgong Xuan Gong added a comment -

            Fix the testcase failures and findbugs issue

            xgong Xuan Gong added a comment - Fix the testcase failures and findbugs issue
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 18s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 17s Maven dependency ordering for branch
            +1 mvninstall 16m 43s trunk passed
            +1 compile 11m 10s trunk passed
            +1 checkstyle 1m 12s trunk passed
            +1 mvnsite 2m 20s trunk passed
            -1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 57s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 1m 51s the patch passed
            +1 compile 6m 36s the patch passed
            +1 javac 6m 36s the patch passed
            -0 checkstyle 1m 7s hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 318 unchanged - 11 fixed = 329 total (was 329)
            +1 mvnsite 2m 13s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 xml 0m 1s The patch has no ill-formed XML file.
            -1 findbugs 1m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
            -1 javadoc 0m 47s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 30s hadoop-yarn-common in the patch passed.
            +1 unit 13m 31s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 4m 56s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 29s The patch does not generate ASF License warnings.
            89m 1s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
              org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.TFileAggregatedLogsBlock.render(HtmlBlock$Block) might ignore java.lang.Exception At TFileAggregatedLogsBlock.java:At TFileAggregatedLogsBlock.java:[line 113]
            Failed junit tests hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestYarnClient
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.api.impl.TestDistributedScheduling



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883787/YARN-6877-trunk.003.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux c157c1780539 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 4b2c442
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/whitespace-eol.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17132/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17132/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 16m 43s trunk passed +1 compile 11m 10s trunk passed +1 checkstyle 1m 12s trunk passed +1 mvnsite 2m 20s trunk passed -1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 57s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 51s the patch passed +1 compile 6m 36s the patch passed +1 javac 6m 36s the patch passed -0 checkstyle 1m 7s hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 318 unchanged - 11 fixed = 329 total (was 329) +1 mvnsite 2m 13s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 1m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 47s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 13m 31s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 4m 56s hadoop-yarn-client in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 89m 1s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common   org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.TFileAggregatedLogsBlock.render(HtmlBlock$Block) might ignore java.lang.Exception At TFileAggregatedLogsBlock.java:At TFileAggregatedLogsBlock.java: [line 113] Failed junit tests hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestYarnClient   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.api.impl.TestDistributedScheduling Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883787/YARN-6877-trunk.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c157c1780539 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b2c442 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17132/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17132/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17132/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            xgong Xuan Gong added a comment -

            Fix the findbugs warning and checkstyle issue.

            I can pass all the testcases on my local machine. Kick the Jenkins again to verify.

            xgong Xuan Gong added a comment - Fix the findbugs warning and checkstyle issue. I can pass all the testcases on my local machine. Kick the Jenkins again to verify.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 16s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 42s Maven dependency ordering for branch
            +1 mvninstall 14m 11s trunk passed
            +1 compile 8m 58s trunk passed
            +1 checkstyle 1m 0s trunk passed
            +1 mvnsite 2m 12s trunk passed
            -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 49s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 36s the patch passed
            +1 compile 5m 36s the patch passed
            +1 javac 5m 36s the patch passed
            -0 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 315 unchanged - 11 fixed = 321 total (was 326)
            +1 mvnsite 2m 7s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 3m 44s the patch passed
            -1 javadoc 0m 42s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 29s hadoop-yarn-common in the patch passed.
            +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 4m 58s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 27s The patch does not generate ASF License warnings.
            81m 39s



            Reason Tests
            Failed junit tests hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestYarnClient



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883986/YARN-6877-trunk.004.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 814466f82dd8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / ae7abad
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17146/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17146/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 14m 11s trunk passed +1 compile 8m 58s trunk passed +1 checkstyle 1m 0s trunk passed +1 mvnsite 2m 12s trunk passed -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 49s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 36s the patch passed +1 compile 5m 36s the patch passed +1 javac 5m 36s the patch passed -0 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 315 unchanged - 11 fixed = 321 total (was 326) +1 mvnsite 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 44s the patch passed -1 javadoc 0m 42s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 29s hadoop-yarn-common in the patch passed. +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 4m 58s hadoop-yarn-client in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 81m 39s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883986/YARN-6877-trunk.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 814466f82dd8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae7abad Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17146/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17146/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 18s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 46s Maven dependency ordering for branch
            +1 mvninstall 16m 13s trunk passed
            +1 compile 10m 50s trunk passed
            +1 checkstyle 1m 6s trunk passed
            +1 mvnsite 2m 25s trunk passed
            -1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 59s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for patch
            +1 mvninstall 1m 52s the patch passed
            +1 compile 6m 46s the patch passed
            +1 javac 6m 46s the patch passed
            -0 checkstyle 1m 1s hadoop-yarn-project/hadoop-yarn: The patch generated 9 new + 315 unchanged - 11 fixed = 324 total (was 326)
            +1 mvnsite 2m 14s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 findbugs 4m 18s the patch passed
            -1 javadoc 0m 43s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 41s hadoop-yarn-common in the patch passed.
            +1 unit 14m 9s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 24s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 5m 42s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 27s The patch does not generate ASF License warnings.
            90m 36s



            Reason Tests
            Failed junit tests hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestYarnClient



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884000/YARN-6877-trunk.005.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 247c58e08d47 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / ae7abad
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/whitespace-eol.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17150/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17150/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests 0 mvndep 0m 46s Maven dependency ordering for branch +1 mvninstall 16m 13s trunk passed +1 compile 10m 50s trunk passed +1 checkstyle 1m 6s trunk passed +1 mvnsite 2m 25s trunk passed -1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 59s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 6m 46s the patch passed +1 javac 6m 46s the patch passed -0 checkstyle 1m 1s hadoop-yarn-project/hadoop-yarn: The patch generated 9 new + 315 unchanged - 11 fixed = 324 total (was 326) +1 mvnsite 2m 14s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 18s the patch passed -1 javadoc 0m 43s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 41s hadoop-yarn-common in the patch passed. +1 unit 14m 9s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 24s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 5m 42s hadoop-yarn-client in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 90m 36s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884000/YARN-6877-trunk.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 247c58e08d47 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae7abad Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17150/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17150/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17150/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            junping_du Junping Du added a comment -

            Thanks xgong for working on a patch.
            The patch is huge. However, mostly are refactoring format binding methods from LogCLIHelpers, LogToolUtils and AggregatedLogsBlock to new created extensible classes, so still straightforward. Some review comments:

            It seems we have two AggregatedLogsBlock now, both are extended from HtmlBlock. One is under filecontroller package (new) while the other in web app package (old):

            +@InterfaceAudience.LimitedPrivate({"YARN", "MapReduce"})
            +public abstract class AggregatedLogsBlock extends HtmlBlock {
            

            What's use case for both AggregatedLogsBlock here? Does old one get dropped/deprecated? If not, we may need to think some other name for the new class to get rid of confusion here.

            Some minor comments:
            In TestLogsCLI.java,

            -    Configuration configuration = new Configuration();
            +    Configuration configuration = new YarnConfiguration();
            

            Why we need this change? I think both should be working here. Do I miss anything?

            In many places, the code are missing space between between '}' and catch

            }catch (NumberFormatException ne) {
            

            In LogCLIHelpers.java, I saw we are mixing using err.println() and System.err.println(). Both way should works fine but better to keep consistent here.

            seems like no change in AggregatedLogsBlockForTest.java. If so, better remove it.

            junping_du Junping Du added a comment - Thanks xgong for working on a patch. The patch is huge. However, mostly are refactoring format binding methods from LogCLIHelpers, LogToolUtils and AggregatedLogsBlock to new created extensible classes, so still straightforward. Some review comments: It seems we have two AggregatedLogsBlock now, both are extended from HtmlBlock. One is under filecontroller package (new) while the other in web app package (old): +@InterfaceAudience.LimitedPrivate({"YARN", "MapReduce"}) +public abstract class AggregatedLogsBlock extends HtmlBlock { What's use case for both AggregatedLogsBlock here? Does old one get dropped/deprecated? If not, we may need to think some other name for the new class to get rid of confusion here. Some minor comments: In TestLogsCLI.java, - Configuration configuration = new Configuration(); + Configuration configuration = new YarnConfiguration(); Why we need this change? I think both should be working here. Do I miss anything? In many places, the code are missing space between between '}' and catch }catch (NumberFormatException ne) { In LogCLIHelpers.java, I saw we are mixing using err.println() and System.err.println(). Both way should works fine but better to keep consistent here. seems like no change in AggregatedLogsBlockForTest.java. If so, better remove it.
            junping_du Junping Du added a comment -

            btw, I just commit YARN-7037 which may need a bit rebase effort here.

            junping_du Junping Du added a comment - btw, I just commit YARN-7037 which may need a bit rebase effort here.
            xgong Xuan Gong added a comment -

            In LogCLIHelpers.java, I saw we are mixing using err.println() and System.err.println(). Both way should works fine but better to keep consistent here.

            This is because some of the function use (PrintStream err) as parameter input, so err.println() will be used. For the functions which do not have (PrintStream err) as parameter, we use System.err.println()

            Uploaded a new patch for other comments.

            xgong Xuan Gong added a comment - In LogCLIHelpers.java, I saw we are mixing using err.println() and System.err.println(). Both way should works fine but better to keep consistent here. This is because some of the function use (PrintStream err) as parameter input, so err.println() will be used. For the functions which do not have (PrintStream err) as parameter, we use System.err.println() Uploaded a new patch for other comments.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 23s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 48s Maven dependency ordering for branch
            +1 mvninstall 13m 37s trunk passed
            +1 compile 8m 53s trunk passed
            +1 checkstyle 1m 1s trunk passed
            +1 mvnsite 2m 16s trunk passed
            -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 59s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 31s the patch passed
            +1 compile 5m 36s the patch passed
            +1 javac 5m 36s the patch passed
            -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 314 unchanged - 11 fixed = 322 total (was 325)
            +1 mvnsite 2m 11s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 3m 45s the patch passed
            -1 javadoc 0m 42s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 40s hadoop-yarn-common in the patch passed.
            +1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 7s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 4m 43s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 31s The patch does not generate ASF License warnings.
            82m 31s



            Reason Tests
            Failed junit tests hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.api.impl.TestYarnClient
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884369/YARN-6877-trunk.006.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 571688f6ed0e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 26fafc3
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/whitespace-eol.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17194/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17194/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 13m 37s trunk passed +1 compile 8m 53s trunk passed +1 checkstyle 1m 1s trunk passed +1 mvnsite 2m 16s trunk passed -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 59s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 5m 36s the patch passed +1 javac 5m 36s the patch passed -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 314 unchanged - 11 fixed = 322 total (was 325) +1 mvnsite 2m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 45s the patch passed -1 javadoc 0m 42s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 40s hadoop-yarn-common in the patch passed. +1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 7s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 4m 43s hadoop-yarn-client in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 82m 31s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.api.impl.TestYarnClient   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884369/YARN-6877-trunk.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 571688f6ed0e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 26fafc3 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17194/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17194/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17194/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            xgong Xuan Gong added a comment -

            All the failed testcases can be passed on my local environment.

            xgong Xuan Gong added a comment - All the failed testcases can be passed on my local environment.
            junping_du Junping Du added a comment -

            Thanks Xuan for uploading a patch!

            Looks like in latest patch, issue still exists for my previous comments:

               public void testFetchApplictionLogsHar() throws Exception {
                 String remoteLogRootDir = "target/logs/";
            -    Configuration configuration = new Configuration();
            +    Configuration configuration = new YarnConfiguration();
            

            Still lacking space between } and catch in some places:

            }catch (NumberFormatException ne) {
            

            Some new comments:

            In LogAggregationWebUtils.java, verifyAndGetLogStartIndex() is better to change to getLogStartIndex(). Normally we should check valid of input if we start with verify*, like original methods - verifyAndGetLogLimits(), but here we just simply throw exceptions (if input is illegal) and rely on caller to catch exception and handle. I think call getLogStartIndex() should be better. Same comments for verifyAndGetLogEndIndex(). But other verfiy* methods should be fine - at least no exception for caller to catch and handle.

            In LogCLIHelpers.dumpAContainerLogsForLogType():

            +      return 0;
            +    } catch (IOException ex) {
            +      System.err.println(ex.getMessage());
                   return -1;
                 }
            

            It looks like we swallow IOException here instead of throw it out as previous behavior. Does this expected? If not, better to keep the original way.

            The same comments for dumpAllContainersLogs().

            I miss one item in previous review for LogAggregationTFileController.java,

            +  @Override
            +  public void closeWriter() {
            +    this.writer.close();
            +  }
            

            Do we need to set writer here to null after we close it which is consistent with previous behavior? The previous intention seems like to keep minimum retain memory. The reason is still valid here. Isn't it?

            Other looks fine to me.

            btw, Findbugs warning shouldn't be related. Can you please check checkstyle, javadoc warnings here?

            junping_du Junping Du added a comment - Thanks Xuan for uploading a patch! Looks like in latest patch, issue still exists for my previous comments: public void testFetchApplictionLogsHar() throws Exception { String remoteLogRootDir = "target/logs/"; - Configuration configuration = new Configuration(); + Configuration configuration = new YarnConfiguration(); Still lacking space between } and catch in some places: }catch (NumberFormatException ne) { Some new comments: In LogAggregationWebUtils.java, verifyAndGetLogStartIndex() is better to change to getLogStartIndex(). Normally we should check valid of input if we start with verify*, like original methods - verifyAndGetLogLimits(), but here we just simply throw exceptions (if input is illegal) and rely on caller to catch exception and handle. I think call getLogStartIndex() should be better. Same comments for verifyAndGetLogEndIndex(). But other verfiy* methods should be fine - at least no exception for caller to catch and handle. In LogCLIHelpers.dumpAContainerLogsForLogType(): + return 0; + } catch (IOException ex) { + System.err.println(ex.getMessage()); return -1; } It looks like we swallow IOException here instead of throw it out as previous behavior. Does this expected? If not, better to keep the original way. The same comments for dumpAllContainersLogs(). I miss one item in previous review for LogAggregationTFileController.java, + @Override + public void closeWriter() { + this.writer.close(); + } Do we need to set writer here to null after we close it which is consistent with previous behavior? The previous intention seems like to keep minimum retain memory. The reason is still valid here. Isn't it? Other looks fine to me. btw, Findbugs warning shouldn't be related. Can you please check checkstyle, javadoc warnings here?
            xgong Xuan Gong added a comment -

            upload a new patch to address all the comments

            xgong Xuan Gong added a comment - upload a new patch to address all the comments
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 29s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 20s Maven dependency ordering for branch
            +1 mvninstall 17m 52s trunk passed
            +1 compile 9m 41s trunk passed
            +1 checkstyle 1m 1s trunk passed
            +1 mvnsite 2m 12s trunk passed
            -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 52s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 40s the patch passed
            +1 compile 6m 2s the patch passed
            +1 javac 6m 2s the patch passed
            -0 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 314 unchanged - 11 fixed = 322 total (was 325)
            +1 mvnsite 2m 14s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 4m 1s the patch passed
            -1 javadoc 0m 44s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 34s hadoop-yarn-common in the patch passed.
            +1 unit 13m 45s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 8s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 5m 14s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 28s The patch does not generate ASF License warnings.
            87m 35s



            Reason Tests
            Failed junit tests hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestYarnClient



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:71bbb86
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884369/YARN-6877-trunk.006.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux a3ba7260f7ed 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / d4417da
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/whitespace-eol.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17234/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17234/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 17m 52s trunk passed +1 compile 9m 41s trunk passed +1 checkstyle 1m 1s trunk passed +1 mvnsite 2m 12s trunk passed -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 52s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 6m 2s the patch passed +1 javac 6m 2s the patch passed -0 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 314 unchanged - 11 fixed = 322 total (was 325) +1 mvnsite 2m 14s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 1s the patch passed -1 javadoc 0m 44s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 34s hadoop-yarn-common in the patch passed. +1 unit 13m 45s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 8s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 5m 14s hadoop-yarn-client in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 87m 35s Reason Tests Failed junit tests hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884369/YARN-6877-trunk.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a3ba7260f7ed 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4417da Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17234/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17234/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17234/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            junping_du Junping Du added a comment -

            Thanks xgong for updating the patch. 007 patch looks good in overall. Only one small NIT here:
            In LogCLIHelpers.dumpAContainerLogsForLogTypeWithoutNodeId() we still catch IOException instead of throw it out.
            Will +1 after fixing this.

            junping_du Junping Du added a comment - Thanks xgong for updating the patch. 007 patch looks good in overall. Only one small NIT here: In LogCLIHelpers.dumpAContainerLogsForLogTypeWithoutNodeId() we still catch IOException instead of throw it out. Will +1 after fixing this.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 21s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for branch
            +1 mvninstall 15m 2s trunk passed
            +1 compile 10m 19s trunk passed
            +1 checkstyle 1m 2s trunk passed
            +1 mvnsite 2m 11s trunk passed
            -1 findbugs 0m 59s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 56s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for patch
            +1 mvninstall 1m 37s the patch passed
            +1 compile 6m 27s the patch passed
            +1 javac 6m 27s the patch passed
            -0 checkstyle 1m 5s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 315 unchanged - 11 fixed = 317 total (was 326)
            +1 mvnsite 2m 8s the patch passed
            +1 whitespace 0m 1s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 findbugs 3m 54s the patch passed
            -1 javadoc 0m 47s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 44s hadoop-yarn-common in the patch passed.
            +1 unit 14m 9s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 10s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 5m 14s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 29s The patch does not generate ASF License warnings.
            86m 23s



            Reason Tests
            Failed junit tests hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestYarnClient



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:71bbb86
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884698/YARN-6877-trunk.007.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 2397dd07a0a1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / d4417da
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17237/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17237/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 15m 2s trunk passed +1 compile 10m 19s trunk passed +1 checkstyle 1m 2s trunk passed +1 mvnsite 2m 11s trunk passed -1 findbugs 0m 59s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 56s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 37s the patch passed +1 compile 6m 27s the patch passed +1 javac 6m 27s the patch passed -0 checkstyle 1m 5s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 315 unchanged - 11 fixed = 317 total (was 326) +1 mvnsite 2m 8s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 54s the patch passed -1 javadoc 0m 47s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 44s hadoop-yarn-common in the patch passed. +1 unit 14m 9s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 10s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 5m 14s hadoop-yarn-client in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 86m 23s Reason Tests Failed junit tests hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884698/YARN-6877-trunk.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2397dd07a0a1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4417da Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17237/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17237/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            xgong Xuan Gong added a comment -

            Thanks for the review. Uploaded a new patch for this

            xgong Xuan Gong added a comment - Thanks for the review. Uploaded a new patch for this
            junping_du Junping Du added a comment -

            Thanks xgong. 008 patch LGTM. +1 pending on Jenkins report.

            junping_du Junping Du added a comment - Thanks xgong . 008 patch LGTM. +1 pending on Jenkins report.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 22m 18s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 48s Maven dependency ordering for branch
            +1 mvninstall 15m 22s trunk passed
            +1 compile 11m 55s trunk passed
            +1 checkstyle 1m 9s trunk passed
            +1 mvnsite 2m 27s trunk passed
            -1 findbugs 0m 58s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 59s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 56s the patch passed
            +1 compile 6m 57s the patch passed
            +1 javac 6m 57s the patch passed
            -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 314 unchanged - 11 fixed = 316 total (was 325)
            +1 mvnsite 2m 30s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 findbugs 4m 16s the patch passed
            -1 javadoc 0m 47s hadoop-yarn-common in the patch failed.
                  Other Tests
            +1 unit 2m 35s hadoop-yarn-common in the patch passed.
            +1 unit 14m 54s hadoop-yarn-server-nodemanager in the patch passed.
            +1 unit 3m 56s hadoop-yarn-server-applicationhistoryservice in the patch passed.
            -1 unit 5m 58s hadoop-yarn-client in the patch failed.
            +1 asflicense 0m 33s The patch does not generate ASF License warnings.
            115m 32s



            Reason Tests
            Failed junit tests hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.api.impl.TestYarnClient
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.TestRMFailover



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:71bbb86
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884791/YARN-6877-trunk.008.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 7e41e0cdcc85 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / d4417da
            Default Java 1.8.0_144
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17243/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17243/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 22m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 15m 22s trunk passed +1 compile 11m 55s trunk passed +1 checkstyle 1m 9s trunk passed +1 mvnsite 2m 27s trunk passed -1 findbugs 0m 58s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 59s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed +1 compile 6m 57s the patch passed +1 javac 6m 57s the patch passed -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 314 unchanged - 11 fixed = 316 total (was 325) +1 mvnsite 2m 30s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 16s the patch passed -1 javadoc 0m 47s hadoop-yarn-common in the patch failed.       Other Tests +1 unit 2m 35s hadoop-yarn-common in the patch passed. +1 unit 14m 54s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 3m 56s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 5m 58s hadoop-yarn-client in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 115m 32s Reason Tests Failed junit tests hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.api.impl.TestYarnClient   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.TestRMFailover Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884791/YARN-6877-trunk.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7e41e0cdcc85 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4417da Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17243/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17243/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            junping_du Junping Du added a comment -

            The test failures (and findbugs warning) shouldn't be related to this patch. Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning. +1. Committing it in.

            junping_du Junping Du added a comment - The test failures (and findbugs warning) shouldn't be related to this patch. Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning. +1. Committing it in.
            junping_du Junping Du added a comment -

            I have commit the patch to trunk. However, trying to backport to branch-2 end up with several file conflicts. xgong, can you upload a patch against branch-2? Thx!

            junping_du Junping Du added a comment - I have commit the patch to trunk. However, trying to backport to branch-2 end up with several file conflicts. xgong , can you upload a patch against branch-2? Thx!
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12287/)
            YARN-6877. Create an abstract log reader for extendability. Contributed (junping_du: rev 91cc070d67533ebb3325b982eba2135e0d175a82)

            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestAppLogAggregatorImpl.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/LogAggregationTFileController.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationHtmlBlock.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/TFileAggregatedLogsBlock.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/package-info.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationWebUtils.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/filecontroller/TestLogAggregationFileControllerFactory.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogToolUtils.java
            • (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationTFileController.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12287/ ) YARN-6877 . Create an abstract log reader for extendability. Contributed (junping_du: rev 91cc070d67533ebb3325b982eba2135e0d175a82) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestAppLogAggregatorImpl.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/LogAggregationTFileController.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationHtmlBlock.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/TFileAggregatedLogsBlock.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/package-info.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationWebUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/filecontroller/TestLogAggregationFileControllerFactory.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogToolUtils.java (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationTFileController.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
            xgong Xuan Gong added a comment - - edited

            junping_du
            Yes, created a patch for branch-2.

            xgong Xuan Gong added a comment - - edited junping_du Yes, created a patch for branch-2.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 25s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  branch-2 Compile Tests
            0 mvndep 0m 25s Maven dependency ordering for branch
            +1 mvninstall 7m 30s branch-2 passed
            +1 compile 2m 14s branch-2 passed with JDK v1.8.0_144
            +1 compile 2m 27s branch-2 passed with JDK v1.7.0_151
            +1 checkstyle 0m 45s branch-2 passed
            +1 mvnsite 2m 2s branch-2 passed
            +1 findbugs 3m 30s branch-2 passed
            +1 javadoc 1m 22s branch-2 passed with JDK v1.8.0_144
            +1 javadoc 1m 28s branch-2 passed with JDK v1.7.0_151
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 43s the patch passed
            +1 compile 2m 4s the patch passed with JDK v1.8.0_144
            +1 javac 2m 4s the patch passed
            +1 compile 2m 31s the patch passed with JDK v1.7.0_151
            +1 javac 2m 31s the patch passed
            -0 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 309 unchanged - 15 fixed = 311 total (was 324)
            +1 mvnsite 1m 57s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 4m 10s the patch passed
            -1 javadoc 0m 25s hadoop-yarn-common in the patch failed with JDK v1.8.0_144.
            -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_151 with JDK v1.7.0_151 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
                  Other Tests
            +1 unit 2m 24s hadoop-yarn-common in the patch passed with JDK v1.7.0_151.
            -1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151.
            +1 unit 3m 18s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_151.
            -1 unit 63m 15s hadoop-yarn-client in the patch failed with JDK v1.7.0_151.
            +1 asflicense 0m 25s The patch does not generate ASF License warnings.
            151m 2s



            Reason Tests
            JDK v1.8.0_144 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
              hadoop.yarn.client.TestApplicationClientProtocolOnHA
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.TestRMFailover
              hadoop.yarn.client.api.impl.TestAMRMClient
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestYarnClient
            JDK v1.7.0_151 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
              hadoop.yarn.client.cli.TestYarnCLI
              hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
              hadoop.yarn.client.cli.TestLogsCLI
              hadoop.yarn.client.api.impl.TestDistributedScheduling
              hadoop.yarn.client.api.impl.TestNMClient
              hadoop.yarn.client.api.impl.TestAMRMProxy
              hadoop.yarn.client.api.impl.TestYarnClient
            JDK v1.7.0_151 Timed out junit tests org.apache.hadoop.yarn.client.TestRMFailover
              org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA
              org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
              org.apache.hadoop.yarn.client.api.impl.TestAMRMClient



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:5e40efe
            JIRA Issue YARN-6877
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884847/YARN-6877-branch-2.002.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux a749ec6f1778 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision branch-2 / 004231d
            Default Java 1.7.0_151
            Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_144.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_151.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_151.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_151.txt
            JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17250/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/17250/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 7m 30s branch-2 passed +1 compile 2m 14s branch-2 passed with JDK v1.8.0_144 +1 compile 2m 27s branch-2 passed with JDK v1.7.0_151 +1 checkstyle 0m 45s branch-2 passed +1 mvnsite 2m 2s branch-2 passed +1 findbugs 3m 30s branch-2 passed +1 javadoc 1m 22s branch-2 passed with JDK v1.8.0_144 +1 javadoc 1m 28s branch-2 passed with JDK v1.7.0_151       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 2m 4s the patch passed with JDK v1.8.0_144 +1 javac 2m 4s the patch passed +1 compile 2m 31s the patch passed with JDK v1.7.0_151 +1 javac 2m 31s the patch passed -0 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 309 unchanged - 15 fixed = 311 total (was 324) +1 mvnsite 1m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 10s the patch passed -1 javadoc 0m 25s hadoop-yarn-common in the patch failed with JDK v1.8.0_144. -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_151 with JDK v1.7.0_151 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)       Other Tests +1 unit 2m 24s hadoop-yarn-common in the patch passed with JDK v1.7.0_151. -1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151. +1 unit 3m 18s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_151. -1 unit 63m 15s hadoop-yarn-client in the patch failed with JDK v1.7.0_151. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 151m 2s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestYarnClient JDK v1.7.0_151 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer   hadoop.yarn.client.cli.TestYarnCLI   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.api.impl.TestDistributedScheduling   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.api.impl.TestYarnClient JDK v1.7.0_151 Timed out junit tests org.apache.hadoop.yarn.client.TestRMFailover   org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884847/YARN-6877-branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a749ec6f1778 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 004231d Default Java 1.7.0_151 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_144.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_151.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_151.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17250/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_151.txt JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17250/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17250/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            junping_du Junping Du added a comment -

            Thanks xgong. The test failures should be unrelated. commit the patch to branch-2 as well.

            junping_du Junping Du added a comment - Thanks xgong . The test failures should be unrelated. commit the patch to branch-2 as well.
            aajisaka Akira Ajisaka added a comment - - edited

            Allen said in dev MLs:

            Looks like YARN-6877 is crashing JDK8 javadoc. Maven stops processing and errors out before even giving a build error/success. Reverting the patch makes things work again. Anyway, Yetus caught it, warned about it continuously, but it was still committed.

            In short, the following warning must not be ignored:

            -1	javadoc	0m 25s	hadoop-yarn-common in the patch failed with JDK v1.8.0_144.
            

            Hi junping_du and xgong, would you fix it?

            aajisaka Akira Ajisaka added a comment - - edited Allen said in dev MLs: Looks like YARN-6877 is crashing JDK8 javadoc. Maven stops processing and errors out before even giving a build error/success. Reverting the patch makes things work again. Anyway, Yetus caught it, warned about it continuously, but it was still committed. In short, the following warning must not be ignored: -1 javadoc 0m 25s hadoop-yarn-common in the patch failed with JDK v1.8.0_144. Hi junping_du and xgong , would you fix it?
            xgong Xuan Gong added a comment -

            ajisakaa

            The branch-2 patch or the trunk patch breaks the build ?

            xgong Xuan Gong added a comment - ajisakaa The branch-2 patch or the trunk patch breaks the build ?
            junping_du Junping Du added a comment -

            Thanks ajisakaa for noticing us.

            In short, the following warning must not be ignored.

            As I mentioned above, I noticed this issue - "Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning." But look at the javadoc warnings link, you will find thousands of warnings/errors there which confuse me that it is just meaning less info.

            Hi Junping Du and Xuan Gong, would you fix it?

            Of course. xgong, can you take a quick look?

            junping_du Junping Du added a comment - Thanks ajisakaa for noticing us. In short, the following warning must not be ignored. As I mentioned above, I noticed this issue - "Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning." But look at the javadoc warnings link, you will find thousands of warnings/errors there which confuse me that it is just meaning less info. Hi Junping Du and Xuan Gong, would you fix it? Of course. xgong , can you take a quick look?
            aajisaka Akira Ajisaka added a comment -

            Thank you for the quick reply!

            The branch-2 patch or the trunk patch breaks the build?

            The both.

            As I mentioned above, I noticed this issue - "Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning." But look at the javadoc warnings link, you will find thousands of warnings/errors there which confuse me that it is just meaning less info.

            Agreed that it is confusing, however, in the report, the command failed by error. For example:

            If there is an error, it fails. Therefore you need to fix all the errors.
            Ideally, all the remaining warnings should be fixed as well to avoid this type of issue.

            aajisaka Akira Ajisaka added a comment - Thank you for the quick reply! The branch-2 patch or the trunk patch breaks the build? The both. As I mentioned above, I noticed this issue - "Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning." But look at the javadoc warnings link, you will find thousands of warnings/errors there which confuse me that it is just meaning less info. Agreed that it is confusing, however, in the report, the command failed by error. For example: hadoop-yarn-common module has some errors, so it failed: https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt hadoop-yarn-client has many warnings but no error, so it succeeded: https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt If there is an error, it fails. Therefore you need to fix all the errors. Ideally, all the remaining warnings should be fixed as well to avoid this type of issue.
            junping_du Junping Du added a comment -

            It is very tricky to find out real errors among thousands of ERROR messages, but I think finally I found two that is really error:

            ...
            [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java:231: error: @param name not found
            [ERROR] * @param the aggregatedLog path.
            ...
            [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java:242: error: @param name not found
            [ERROR] * @param the aggregatedLog path.
            

            xgong, can you double check any other java doc errors?

            junping_du Junping Du added a comment - It is very tricky to find out real errors among thousands of ERROR messages, but I think finally I found two that is really error: ... [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java:231: error: @param name not found [ERROR] * @param the aggregatedLog path. ... [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java:242: error: @param name not found [ERROR] * @param the aggregatedLog path. xgong , can you double check any other java doc errors?
            aajisaka Akira Ajisaka added a comment -

            Thanks, I think you are right. Grepping "error:" works.

            aajisaka Akira Ajisaka added a comment - Thanks, I think you are right. Grepping "error:" works.
            xgong Xuan Gong added a comment - - edited

            junping_du ajisakaa

            Those log aggregation related errors will be fixed with https://issues.apache.org/jira/browse/YARN-7148 which I have already uploaded a patch.

            xgong Xuan Gong added a comment - - edited junping_du ajisakaa Those log aggregation related errors will be fixed with https://issues.apache.org/jira/browse/YARN-7148 which I have already uploaded a patch.

            People

              xgong Xuan Gong
              xgong Xuan Gong
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: