Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: timelineserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In this ticket, we will add new put APIs in timelineClient to let clients/applications have the option to use ATS v1.5

      1. YARN-4234.addendum.patch
        11 kB
        Masatake Iwasaki
      2. YARN-4234.2015-12-21.1.patch
        82 kB
        Xuan Gong
      3. YARN-4234.2015-12-18.1.patch
        81 kB
        Xuan Gong
      4. YARN-4234.2015-12-18.patch
        82 kB
        Xuan Gong
      5. YARN-4234.2015-11-18.patch
        82 kB
        Xuan Gong
      6. YARN-4234.2015-12-17.1.patch
        81 kB
        Xuan Gong
      7. YARN-4234.2015-12-09.patch
        78 kB
        Xuan Gong
      8. YARN-4234.2015-12-09.patch
        77 kB
        Xuan Gong
      9. YARN-4234.2015-11-18.2.patch
        77 kB
        Xuan Gong
      10. YARN-4234.2015-11-18.1.patch
        77 kB
        Xuan Gong
      11. YARN-4234-2015-11-16.2.patch
        72 kB
        Xuan Gong
      12. YARN-4234-2015-11-16.1.patch
        71 kB
        Xuan Gong
      13. YARN-4234-2015-11-13.1.patch
        71 kB
        Xuan Gong
      14. YARN-4234.2015-11-12.1.patch
        44 kB
        Xuan Gong
      15. YARN-4234.2015-11-12.1.patch
        44 kB
        Xuan Gong
      16. YARN-4234-20151111.2.patch
        44 kB
        Xuan Gong
      17. YARN-4234.20151111.1.patch
        48 kB
        Xuan Gong
      18. YARN-4234.20151110.1.patch
        44 kB
        Xuan Gong
      19. YARN-4234.20151109.patch
        43 kB
        Xuan Gong
      20. YARN-4234.3.patch
        20 kB
        Xuan Gong
      21. YARN-4234.2.patch
        20 kB
        Xuan Gong
      22. YARN-4234.1.patch
        19 kB
        Xuan Gong

        Issue Links

          Activity

          Hide
          djp Junping Du added a comment -

          Thanks all for fixing this.

          Show
          djp Junping Du added a comment - Thanks all for fixing this.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          sorry, I just forgot to push. thanks for following up!

          Show
          ozawa Tsuyoshi Ozawa added a comment - sorry, I just forgot to push. thanks for following up!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          committed addendum patch by Masatake to trunk and branch-2(Just removing a file "q" in root directory). Thanks for your contribution!

          Cherry-picked this to branch-2.8 as well. Thanks Tsuyoshi and Masatake.

          Show
          ajisakaa Akira Ajisaka added a comment - committed addendum patch by Masatake to trunk and branch-2(Just removing a file "q" in root directory). Thanks for your contribution! Cherry-picked this to branch-2.8 as well. Thanks Tsuyoshi and Masatake.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9023 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9023/)
          YARN-4234. addendum patch to remove unnecessary file. Contributed by (ozawa: rev 65537845a9799c6a73eecccb4cd437def58459d8)

          • q
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9023 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9023/ ) YARN-4234 . addendum patch to remove unnecessary file. Contributed by (ozawa: rev 65537845a9799c6a73eecccb4cd437def58459d8) q
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Junping Du Masatake Iwasaki committed addendum patch by Masatake to trunk and branch-2(Just removing a file "q" in root directory). Thanks for your contribution!

          Show
          ozawa Tsuyoshi Ozawa added a comment - Junping Du Masatake Iwasaki committed addendum patch by Masatake to trunk and branch-2(Just removing a file "q" in root directory). Thanks for your contribution!
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Masatake Iwasaki thanks for following up. +1, checking this in.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Masatake Iwasaki thanks for following up. +1, checking this in.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Junping Du, file named "q" seemed to be accidentally added to top directory. I'm adding addendum patch to remove the file.

          Show
          iwasakims Masatake Iwasaki added a comment - Junping Du , file named "q" seemed to be accidentally added to top directory. I'm adding addendum patch to remove the file.
          Hide
          djp Junping Du added a comment -

          Forget to mention, I think we need to document new APIs that we are adding here. Just filed YARN-4501 to track this effort.

          Show
          djp Junping Du added a comment - Forget to mention, I think we need to document new APIs that we are adding here. Just filed YARN-4501 to track this effort.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9018/)
          YARN-4234. New put APIs in TimelineClient for ats v1.5. Contributed by (junping_du: rev 882f2f04644a13cadb93070d5545f7a4f8691fde)

          • q
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientForATS1_5.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineWriter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelineEntityGroupId.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestTimelineEntityGroupId.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/DirectTimelineWriter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServicesWithSSL.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9018/ ) YARN-4234 . New put APIs in TimelineClient for ats v1.5. Contributed by (junping_du: rev 882f2f04644a13cadb93070d5545f7a4f8691fde) q hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientForATS1_5.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineWriter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelineEntityGroupId.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestTimelineEntityGroupId.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/DirectTimelineWriter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServicesWithSSL.java
          Hide
          djp Junping Du added a comment -

          I have commit the latest the patch to trunk, branch-2 and branch-2.8. Thanks Xuan for contributing the patch and Vinod, Li, Jason and Naga for review and comments!

          Show
          djp Junping Du added a comment - I have commit the latest the patch to trunk, branch-2 and branch-2.8. Thanks Xuan for contributing the patch and Vinod, Li, Jason and Naga for review and comments!
          Hide
          djp Junping Du added a comment -

          Thanks Xuan Gong for updating the patch. +1 on latest patch. I will commit it tomorrow if no further comments or objections from others.

          Show
          djp Junping Du added a comment - Thanks Xuan Gong for updating the patch. +1 on latest patch. I will commit it tomorrow if no further comments or objections from others.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 38s trunk passed
          +1 compile 2m 11s trunk passed with JDK v1.8.0_66
          +1 compile 2m 25s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 40s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 3m 41s trunk passed
          +1 javadoc 1m 32s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 59s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 31s the patch passed
          +1 compile 2m 11s the patch passed with JDK v1.8.0_66
          +1 javac 2m 11s the patch passed
          +1 compile 2m 32s the patch passed with JDK v1.7.0_91
          +1 javac 2m 32s the patch passed
          -1 checkstyle 0m 29s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 237).
          +1 mvnsite 1m 38s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 1s the patch passed
          +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 3s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 12s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 4m 17s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 25s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 4m 16s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          60m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778942/YARN-4234.2015-12-21.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 62a04ec61dbf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a0de702
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10065/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10065/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10065/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 38s trunk passed +1 compile 2m 11s trunk passed with JDK v1.8.0_66 +1 compile 2m 25s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 40s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 3m 41s trunk passed +1 javadoc 1m 32s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 59s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 31s the patch passed +1 compile 2m 11s the patch passed with JDK v1.8.0_66 +1 javac 2m 11s the patch passed +1 compile 2m 32s the patch passed with JDK v1.7.0_91 +1 javac 2m 32s the patch passed -1 checkstyle 0m 29s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 237). +1 mvnsite 1m 38s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 1s the patch passed +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 3s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 12s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 4m 17s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 25s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 4m 16s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 60m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778942/YARN-4234.2015-12-21.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 62a04ec61dbf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0de702 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10065/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10065/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10065/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 3s trunk passed
          +1 compile 2m 8s trunk passed with JDK v1.8.0_66
          +1 compile 2m 22s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 48s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 21s the patch passed
          +1 compile 2m 5s the patch passed with JDK v1.8.0_66
          +1 javac 2m 5s the patch passed
          +1 compile 2m 16s the patch passed with JDK v1.7.0_91
          +1 javac 2m 16s the patch passed
          -1 checkstyle 0m 28s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 237).
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 52s the patch passed
          +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 46s the patch passed with JDK v1.7.0_91
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 44s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 17s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 3m 57s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          56m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778942/YARN-4234.2015-12-21.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78c8ee747cbb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a0de702
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10064/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10064/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10064/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 3s trunk passed +1 compile 2m 8s trunk passed with JDK v1.8.0_66 +1 compile 2m 22s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 21s the patch passed +1 compile 2m 5s the patch passed with JDK v1.8.0_66 +1 javac 2m 5s the patch passed +1 compile 2m 16s the patch passed with JDK v1.7.0_91 +1 javac 2m 16s the patch passed -1 checkstyle 0m 28s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 237). +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 52s the patch passed +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 46s the patch passed with JDK v1.7.0_91 +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 3m 44s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 17s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 3m 57s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 56m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778942/YARN-4234.2015-12-21.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78c8ee747cbb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0de702 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10064/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10064/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10064/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Once this patch is committed, would anyone of you please ping me so that I can quickly rebase YARN-4265?

          I think you will receive notification automatically if you watch on this JIRA.

          Show
          djp Junping Du added a comment - Once this patch is committed, would anyone of you please ping me so that I can quickly rebase YARN-4265 ? I think you will receive notification automatically if you watch on this JIRA.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks for the work Xuan Gong and Junping Du! Once this patch is committed, would anyone of you please ping me so that I can quickly rebase YARN-4265?

          Show
          gtCarrera9 Li Lu added a comment - Thanks for the work Xuan Gong and Junping Du ! Once this patch is committed, would anyone of you please ping me so that I can quickly rebase YARN-4265 ?
          Hide
          xgong Xuan Gong added a comment -

          Uploaded a new patch to fix checkstyle warning

          Show
          xgong Xuan Gong added a comment - Uploaded a new patch to fix checkstyle warning
          Hide
          djp Junping Du added a comment -

          Hi Xuan, thanks for updating the patch! There are still some checksyte issues reported related to your patch. I know some of them are invalid, like "File length is 2,385 lines" or "More than 7 parameters", but some is valid like naming issue for "timelineEntityGroupIdStrPrefix". Can you fix the left issues? We are getting quite closed now. Thanks for the patience.

          Show
          djp Junping Du added a comment - Hi Xuan, thanks for updating the patch! There are still some checksyte issues reported related to your patch. I know some of them are invalid, like "File length is 2,385 lines" or "More than 7 parameters", but some is valid like naming issue for "timelineEntityGroupIdStrPrefix". Can you fix the left issues? We are getting quite closed now. Thanks for the patience.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 7s trunk passed
          +1 compile 2m 10s trunk passed with JDK v1.8.0_66
          +1 compile 2m 19s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 56s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 23s the patch passed
          +1 compile 2m 9s the patch passed with JDK v1.8.0_66
          +1 javac 2m 9s the patch passed
          +1 compile 2m 21s the patch passed with JDK v1.7.0_91
          +1 javac 2m 21s the patch passed
          -1 checkstyle 0m 31s Patch generated 12 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 246).
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 55s the patch passed
          +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 55s the patch passed with JDK v1.7.0_91
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 54s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 20s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 4m 7s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          57m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778599/YARN-4234.2015-12-18.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 630ab8db90d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e63388f
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10045/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10045/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10045/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 7s trunk passed +1 compile 2m 10s trunk passed with JDK v1.8.0_66 +1 compile 2m 19s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 56s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 23s the patch passed +1 compile 2m 9s the patch passed with JDK v1.8.0_66 +1 javac 2m 9s the patch passed +1 compile 2m 21s the patch passed with JDK v1.7.0_91 +1 javac 2m 21s the patch passed -1 checkstyle 0m 31s Patch generated 12 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 246). +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 55s the patch passed +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 55s the patch passed with JDK v1.7.0_91 +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 3m 54s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 20s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 4m 7s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 57m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778599/YARN-4234.2015-12-18.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 630ab8db90d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e63388f findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10045/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10045/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10045/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          upload a patch to fix the checkstyle issue

          Show
          xgong Xuan Gong added a comment - upload a patch to fix the checkstyle issue
          Hide
          djp Junping Du added a comment -

          Thanks Xuan Gong for updating the patch. The latest patch LGTM in overall. However, it seems to be many checkstyle issues reported by Jenkins are related to this patch. May be we should fix them before we get patch in?

          Show
          djp Junping Du added a comment - Thanks Xuan Gong for updating the patch. The latest patch LGTM in overall. However, it seems to be many checkstyle issues reported by Jenkins are related to this patch. May be we should fix them before we get patch in?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 8m 12s trunk passed
          +1 compile 2m 12s trunk passed with JDK v1.8.0_66
          +1 compile 2m 19s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 2m 0s trunk passed
          +1 mvneclipse 0m 54s trunk passed
          +1 findbugs 4m 30s trunk passed
          +1 javadoc 1m 47s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 23s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 47s the patch passed
          +1 compile 2m 6s the patch passed with JDK v1.8.0_66
          +1 javac 2m 6s the patch passed
          +1 compile 2m 18s the patch passed with JDK v1.7.0_91
          +1 javac 2m 18s the patch passed
          -1 checkstyle 0m 29s Patch generated 67 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 301).
          +1 mvnsite 2m 1s the patch passed
          +1 mvneclipse 0m 54s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 0s The patch has 4 line(s) with tabs.
          +1 findbugs 4m 59s the patch passed
          +1 javadoc 1m 47s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 12s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 2s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 47s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 17s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 14s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 unit 3m 19s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          78m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778362/YARN-4234.2015-12-17.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b59dc7f0c619 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1de56b0
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10029/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10029/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10029/artifact/patchprocess/whitespace-tabs.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10029/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10029/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 8m 12s trunk passed +1 compile 2m 12s trunk passed with JDK v1.8.0_66 +1 compile 2m 19s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 0s trunk passed +1 mvneclipse 0m 54s trunk passed +1 findbugs 4m 30s trunk passed +1 javadoc 1m 47s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 23s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 47s the patch passed +1 compile 2m 6s the patch passed with JDK v1.8.0_66 +1 javac 2m 6s the patch passed +1 compile 2m 18s the patch passed with JDK v1.7.0_91 +1 javac 2m 18s the patch passed -1 checkstyle 0m 29s Patch generated 67 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 301). +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 54s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 0s The patch has 4 line(s) with tabs. +1 findbugs 4m 59s the patch passed +1 javadoc 1m 47s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 12s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 2s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 47s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 17s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 14s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 unit 3m 19s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 78m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778362/YARN-4234.2015-12-17.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b59dc7f0c619 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1de56b0 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10029/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10029/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10029/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10029/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10029/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          Thanks for the review. Junping Du. Attached a new patch to address all your comments

          Show
          xgong Xuan Gong added a comment - Thanks for the review. Junping Du . Attached a new patch to address all your comments
          Hide
          djp Junping Du added a comment -

          Thanks Xuan Gong for updating the patch. I just go through the latest patch, and below is my review comments:

          In TimelineEntityGroupId.java,
          Can we add Javadoc for TimelineEntityGroupId and explain what TimelineEntityGroupId is used for?
          Also in TimelineEntityGroupId.equals(), can we omit unnecessary ";"?

          In FileSystemTimelineWriter.java,

                LOG.debug(
                    YarnConfiguration.TIMELINE_SERVICE_CLIENT_FD_FLUSH_INTERVAL_SECS
                        + "=" + flushIntervalSecs + ", " +
                    YarnConfiguration.TIMELINE_SERVICE_CLIENT_FD_CLEAN_INTERVAL_SECS
                        + "=" + cleanIntervalSecs + ", " +
                    YarnConfiguration.TIMELINE_SERVICE_CLIENT_FD_RETAIN_SECS
                        + "=" + ttl + ", " +
                    TIMELINE_SERVICE_ENTITYFILE_FS_SUPPORT_APPEND
                        + "=" + isAppendSupported);
          

          Shall we log other related configurations, like: activePath, retryPolicy, summaryEntityTypes, etc.?

          In putEntities(),
          The name of entitiesToSummary, entitiesToEntity and entitiesToDB sounds a little confusing. How about update them to entitiesToSummaryCache, entitiesToEntityCache and entitiesToDBStore? Also, we shouldn't log in INFO level for each entity get written in cache. Debug level should be good.

          In LogFDsCache,

                summanyLogFDs = new HashMap<ApplicationAttemptId, EntityLogFD>();
                entityLogFDs = new HashMap<ApplicationAttemptId,
                    HashMap<TimelineEntityGroupId, EntityLogFD>>();
          

          There is a race condition here: if LogFDsCache is doing flush with iterating the map and some writing entity/summary come to insert the HashMap at the same time, then it will throw ConcurrentModificationException. We should use ConcurrentHashMap instead.
          Also, for CleanInActiveFDsTask, We should use WARN level for LOG instead of DEBUG in case exception get throw.

          In TimelineWriter.java,
          The conf seems not get used. If so, we can remove it from class and constructor?

          Other looks good to me.

          Show
          djp Junping Du added a comment - Thanks Xuan Gong for updating the patch. I just go through the latest patch, and below is my review comments: In TimelineEntityGroupId.java, Can we add Javadoc for TimelineEntityGroupId and explain what TimelineEntityGroupId is used for? Also in TimelineEntityGroupId.equals(), can we omit unnecessary ";"? In FileSystemTimelineWriter.java, LOG.debug( YarnConfiguration.TIMELINE_SERVICE_CLIENT_FD_FLUSH_INTERVAL_SECS + "=" + flushIntervalSecs + ", " + YarnConfiguration.TIMELINE_SERVICE_CLIENT_FD_CLEAN_INTERVAL_SECS + "=" + cleanIntervalSecs + ", " + YarnConfiguration.TIMELINE_SERVICE_CLIENT_FD_RETAIN_SECS + "=" + ttl + ", " + TIMELINE_SERVICE_ENTITYFILE_FS_SUPPORT_APPEND + "=" + isAppendSupported); Shall we log other related configurations, like: activePath, retryPolicy, summaryEntityTypes, etc.? In putEntities(), The name of entitiesToSummary, entitiesToEntity and entitiesToDB sounds a little confusing. How about update them to entitiesToSummaryCache, entitiesToEntityCache and entitiesToDBStore? Also, we shouldn't log in INFO level for each entity get written in cache. Debug level should be good. In LogFDsCache, summanyLogFDs = new HashMap<ApplicationAttemptId, EntityLogFD>(); entityLogFDs = new HashMap<ApplicationAttemptId, HashMap<TimelineEntityGroupId, EntityLogFD>>(); There is a race condition here: if LogFDsCache is doing flush with iterating the map and some writing entity/summary come to insert the HashMap at the same time, then it will throw ConcurrentModificationException. We should use ConcurrentHashMap instead. Also, for CleanInActiveFDsTask, We should use WARN level for LOG instead of DEBUG in case exception get throw. In TimelineWriter.java, The conf seems not get used. If so, we can remove it from class and constructor? Other looks good to me.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 28s trunk passed
          +1 compile 2m 16s trunk passed with JDK v1.8.0_66
          +1 compile 2m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 3m 24s trunk passed
          +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 42s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 20s the patch passed
          +1 compile 2m 1s the patch passed with JDK v1.8.0_66
          +1 javac 2m 1s the patch passed
          +1 compile 2m 19s the patch passed with JDK v1.7.0_91
          +1 javac 2m 19s the patch passed
          -1 checkstyle 0m 28s Patch generated 53 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 287).
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 54s the patch passed
          +1 javadoc 1m 31s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 47s the patch passed with JDK v1.7.0_91
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 43s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 18s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 3m 57s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          56m 31s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776649/YARN-4234.2015-12-09.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b4d5f179a9c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f910e4f
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9929/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9929/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9929/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 28s trunk passed +1 compile 2m 16s trunk passed with JDK v1.8.0_66 +1 compile 2m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 3m 24s trunk passed +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 42s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 20s the patch passed +1 compile 2m 1s the patch passed with JDK v1.8.0_66 +1 javac 2m 1s the patch passed +1 compile 2m 19s the patch passed with JDK v1.7.0_91 +1 javac 2m 19s the patch passed -1 checkstyle 0m 28s Patch generated 53 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 237, now 287). +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 54s the patch passed +1 javadoc 1m 31s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 47s the patch passed with JDK v1.7.0_91 +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 3m 43s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 18s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 3m 57s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 56m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776649/YARN-4234.2015-12-09.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b4d5f179a9c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f910e4f findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9929/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9929/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9929/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          Patch generally LGTM. The only issue is that if NameNode is unavailable and retry is not set, the timeline client will quickly retry and then fail. This will cause either application attempts to fail, or the RM to fail to start. Maybe we can try some mechanisms like in FileSystemRMStateStore#startInternal, where we explicitly change related retry policy config?

          Other than this corner case issue I'm fine with this patch. Right now people are reaching agreements on YARN-3623, so probably YARN-3623 can go in very soon. This said, could some committers please review the current patch? Thanks!

          Show
          gtCarrera9 Li Lu added a comment - Patch generally LGTM. The only issue is that if NameNode is unavailable and retry is not set, the timeline client will quickly retry and then fail. This will cause either application attempts to fail, or the RM to fail to start. Maybe we can try some mechanisms like in FileSystemRMStateStore#startInternal, where we explicitly change related retry policy config? Other than this corner case issue I'm fine with this patch. Right now people are reaching agreements on YARN-3623 , so probably YARN-3623 can go in very soon. This said, could some committers please review the current patch? Thanks!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          As I commented on YARN-3623, we should split this patch and put the common code at YARN-3623.

          Once we do it, this JIRA depends on YARN-3623, linking the tickets to reflect that.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - As I commented on YARN-3623 , we should split this patch and put the common code at YARN-3623 . Once we do it, this JIRA depends on YARN-3623 , linking the tickets to reflect that.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 9m 31s trunk passed
          +1 compile 2m 52s trunk passed with JDK v1.8.0_66
          +1 compile 2m 46s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 45s trunk passed
          +1 mvneclipse 0m 49s trunk passed
          -1 findbugs 1m 38s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 54s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 18s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 39s the patch passed
          +1 compile 2m 54s the patch passed with JDK v1.8.0_66
          +1 javac 2m 54s the patch passed
          +1 compile 2m 44s the patch passed with JDK v1.7.0_85
          +1 javac 2m 44s the patch passed
          -1 checkstyle 0m 33s Patch generated 56 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 288).
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 38s the patch passed
          +1 javadoc 1m 53s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 23s the patch passed with JDK v1.7.0_85
          +1 unit 0m 32s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 35s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 40s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 32s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 34s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 3m 38s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          65m 43s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-19
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773145/YARN-4234.2015-11-18.2.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c387bcc05c63 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / 23a130a
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9734/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9734/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9734/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 77MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9734/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 9m 31s trunk passed +1 compile 2m 52s trunk passed with JDK v1.8.0_66 +1 compile 2m 46s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 45s trunk passed +1 mvneclipse 0m 49s trunk passed -1 findbugs 1m 38s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 54s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 18s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 39s the patch passed +1 compile 2m 54s the patch passed with JDK v1.8.0_66 +1 javac 2m 54s the patch passed +1 compile 2m 44s the patch passed with JDK v1.7.0_85 +1 javac 2m 44s the patch passed -1 checkstyle 0m 33s Patch generated 56 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 288). +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 38s the patch passed +1 javadoc 1m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 23s the patch passed with JDK v1.7.0_85 +1 unit 0m 32s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 35s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 3m 40s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 32s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 34s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 3m 38s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 65m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-19 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773145/YARN-4234.2015-11-18.2.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c387bcc05c63 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 23a130a findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9734/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9734/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9734/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 77MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9734/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 11s trunk passed
          +1 compile 2m 7s trunk passed with JDK v1.8.0_66
          +1 compile 2m 17s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          -1 findbugs 1m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 54s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 20s the patch passed
          +1 compile 2m 4s the patch passed with JDK v1.8.0_66
          +1 javac 2m 4s the patch passed
          +1 compile 2m 19s the patch passed with JDK v1.7.0_85
          +1 javac 2m 19s the patch passed
          -1 checkstyle 0m 28s Patch generated 56 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 288).
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common introduced 1 new FindBugs issues.
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 39s the patch passed with JDK v1.7.0_85
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 55s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 3m 9s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          54m 0s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
            Inconsistent synchronization of org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter$LogFD.lastModifiedTime; locked 50% of time Unsynchronized access at FileSystemTimelineWriter.java:50% of time Unsynchronized access at FileSystemTimelineWriter.java:[line 340]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-18
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773073/YARN-4234.2015-11-18.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux edcfc020ce69 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / bad2afe
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9725/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9725/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9725/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9725/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 78MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9725/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 11s trunk passed +1 compile 2m 7s trunk passed with JDK v1.8.0_66 +1 compile 2m 17s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 40s trunk passed -1 findbugs 1m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 54s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 20s the patch passed +1 compile 2m 4s the patch passed with JDK v1.8.0_66 +1 javac 2m 4s the patch passed +1 compile 2m 19s the patch passed with JDK v1.7.0_85 +1 javac 2m 19s the patch passed -1 checkstyle 0m 28s Patch generated 56 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 288). +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common introduced 1 new FindBugs issues. +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 39s the patch passed with JDK v1.7.0_85 +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 2m 55s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 3m 9s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 54m 0s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common   Inconsistent synchronization of org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter$LogFD.lastModifiedTime; locked 50% of time Unsynchronized access at FileSystemTimelineWriter.java:50% of time Unsynchronized access at FileSystemTimelineWriter.java: [line 340] Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-18 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773073/YARN-4234.2015-11-18.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux edcfc020ce69 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / bad2afe findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9725/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9725/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9725/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9725/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 78MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9725/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          The new patch addressed the comments

          Show
          xgong Xuan Gong added a comment - The new patch addressed the comments
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi Xuan Gong, the current fix will force to close the current FD if the incoming write request is to a different entity group. This may generate a lot of files since users may switch to different entity groups arbitrarily. Meanwhile, we're changing the content of the concurrent hash table, but not enforcing strong consistency. Maybe we'd like to switch the hash map to use entity group id for each FDs?

          Show
          gtCarrera9 Li Lu added a comment - Hi Xuan Gong , the current fix will force to close the current FD if the incoming write request is to a different entity group. This may generate a lot of files since users may switch to different entity groups arbitrarily. Meanwhile, we're changing the content of the concurrent hash table, but not enforcing strong consistency. Maybe we'd like to switch the hash map to use entity group id for each FDs?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 1m 11s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 3m 20s trunk passed
          +1 compile 1m 2s trunk passed with JDK v1.8.0_66
          +1 compile 1m 0s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          -1 findbugs 1m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 35s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 5s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 17s the patch passed
          +1 compile 1m 0s the patch passed with JDK v1.8.0_66
          +1 javac 1m 0s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_85
          +1 javac 0m 57s the patch passed
          -1 checkstyle 0m 30s Patch generated 41 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 273).
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 8s the patch passed
          +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 0s the patch passed with JDK v1.7.0_85
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 56s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 22s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 4m 7s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          49m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-17
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772626/YARN-4234-2015-11-16.2.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 54e80e4bab55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-18bd53c/precommit/personality/hadoop.sh
          git revision trunk / 6502d59
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9708/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9708/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9708/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9708/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 11s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 3m 20s trunk passed +1 compile 1m 2s trunk passed with JDK v1.8.0_66 +1 compile 1m 0s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 44s trunk passed -1 findbugs 1m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 35s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 5s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 17s the patch passed +1 compile 1m 0s the patch passed with JDK v1.8.0_66 +1 javac 1m 0s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_85 +1 javac 0m 57s the patch passed -1 checkstyle 0m 30s Patch generated 41 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 273). +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 8s the patch passed +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 0s the patch passed with JDK v1.7.0_85 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 3m 56s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 22s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 4m 7s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 49m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-17 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772626/YARN-4234-2015-11-16.2.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 54e80e4bab55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-18bd53c/precommit/personality/hadoop.sh git revision trunk / 6502d59 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9708/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9708/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9708/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9708/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          While this is totally fine for summary logs, it will cause entities belong to different entity groups to be redirected to the wrong file.

          Right.. The new patch addressed this issue. Also added a testcase to test this

          Show
          xgong Xuan Gong added a comment - While this is totally fine for summary logs, it will cause entities belong to different entity groups to be redirected to the wrong file. Right.. The new patch addressed this issue. Also added a testcase to test this
          Hide
          gtCarrera9 Li Lu added a comment -

          One more problem: Looking into the writer, I can see the writer maintains one file descriptor per app attempt:
          private Map<ApplicationAttemptId, EntityLogFD> entityLogFDs;
          While this is totally fine for summary logs, it will cause entities belong to different entity groups to be redirected to the wrong file.
          Maybe we need to maintain a mapping between active entity groups (will there be too many?) to opened file descriptors? In this way we can find the right file to write for each entity group id.

          Show
          gtCarrera9 Li Lu added a comment - One more problem: Looking into the writer, I can see the writer maintains one file descriptor per app attempt: private Map<ApplicationAttemptId, EntityLogFD> entityLogFDs; While this is totally fine for summary logs, it will cause entities belong to different entity groups to be redirected to the wrong file. Maybe we need to maintain a mapping between active entity groups (will there be too many?) to opened file descriptors? In this way we can find the right file to write for each entity group id.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 3m 27s trunk passed
          +1 compile 0m 50s trunk passed with JDK v1.8.0_66
          +1 compile 0m 48s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 18s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 36s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 9s the patch passed
          +1 compile 0m 48s the patch passed with JDK v1.8.0_66
          +1 javac 0m 48s the patch passed
          +1 compile 0m 48s the patch passed with JDK v1.7.0_85
          +1 javac 0m 48s the patch passed
          -1 checkstyle 0m 25s Patch generated 41 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 273).
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 33s the patch passed
          +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 30s the patch passed with JDK v1.7.0_85
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 49s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 52s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          41m 23s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772566/YARN-4234-2015-11-16.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7bc4f480da1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / 02653ad
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9698/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9698/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9698/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9698/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 3m 27s trunk passed +1 compile 0m 50s trunk passed with JDK v1.8.0_66 +1 compile 0m 48s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 41s trunk passed -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 18s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 36s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 9s the patch passed +1 compile 0m 48s the patch passed with JDK v1.8.0_66 +1 javac 0m 48s the patch passed +1 compile 0m 48s the patch passed with JDK v1.7.0_85 +1 javac 0m 48s the patch passed -1 checkstyle 0m 25s Patch generated 41 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 273). +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 33s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 30s the patch passed with JDK v1.7.0_85 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 49s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 2m 52s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 3m 6s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 41m 23s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772566/YARN-4234-2015-11-16.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7bc4f480da1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / 02653ad findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9698/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9698/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9698/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9698/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi Xuan Gong, not sure if you've noticed Sangjin Lee's comments in YATN-4183:

          (yarn.timeline-service.version)

          I'd like to point out an interesting possibility raised in another JIRA by Joep Rottinghuis. With v.2 and especially early on with v.2, it would be rather useful to be able to enable both v.1 (or v.1.5) and v.2. That would provide a useful verification and comparison environment with a single cluster. The way it's being discussed right now, it sounds like the version would be a single value (mutually exclusive). Wouldn't it be good to have a possibility to be able to enable more than one version? Thoughts?

          I think the proposal is still compatible with the current design here, with some simple changes on the sanity check side. How about implement this proposal? Thanks!

          Show
          gtCarrera9 Li Lu added a comment - Hi Xuan Gong , not sure if you've noticed Sangjin Lee 's comments in YATN-4183: (yarn.timeline-service.version) I'd like to point out an interesting possibility raised in another JIRA by Joep Rottinghuis. With v.2 and especially early on with v.2, it would be rather useful to be able to enable both v.1 (or v.1.5) and v.2. That would provide a useful verification and comparison environment with a single cluster. The way it's being discussed right now, it sounds like the version would be a single value (mutually exclusive). Wouldn't it be good to have a possibility to be able to enable more than one version? Thoughts? I think the proposal is still compatible with the current design here, with some simple changes on the sanity check side. How about implement this proposal? Thanks!
          Hide
          xgong Xuan Gong added a comment -

          Fix the testcase failures and whitespace warning.

          Show
          xgong Xuan Gong added a comment - Fix the testcase failures and whitespace warning.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 3m 21s trunk passed
          +1 compile 0m 56s trunk passed with JDK v1.8.0_60
          +1 compile 0m 50s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 18s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          -1 findbugs 1m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 27s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 51s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 14s the patch passed
          +1 compile 0m 54s the patch passed with JDK v1.8.0_60
          +1 javac 0m 54s the patch passed
          +1 compile 0m 51s the patch passed with JDK v1.7.0_79
          +1 javac 0m 51s the patch passed
          -1 checkstyle 0m 27s Patch generated 44 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 276).
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 3m 47s the patch passed
          +1 javadoc 1m 26s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 53s the patch passed with JDK v1.7.0_79
          -1 unit 0m 22s hadoop-yarn-api in the patch failed with JDK v1.8.0_60.
          +1 unit 2m 0s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          +1 unit 3m 41s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_60.
          -1 unit 0m 23s hadoop-yarn-api in the patch failed with JDK v1.7.0_79.
          +1 unit 2m 13s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          +1 unit 3m 52s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          45m 22s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
          JDK v1.7.0_79 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772445/YARN-4234-2015-11-13.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 94f484aac299 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / 855d529
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9693/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9693/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 3m 21s trunk passed +1 compile 0m 56s trunk passed with JDK v1.8.0_60 +1 compile 0m 50s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 18s trunk passed +1 mvneclipse 0m 41s trunk passed -1 findbugs 1m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 27s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 51s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 14s the patch passed +1 compile 0m 54s the patch passed with JDK v1.8.0_60 +1 javac 0m 54s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_79 +1 javac 0m 51s the patch passed -1 checkstyle 0m 27s Patch generated 44 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 276). +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 41s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 47s the patch passed +1 javadoc 1m 26s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 53s the patch passed with JDK v1.7.0_79 -1 unit 0m 22s hadoop-yarn-api in the patch failed with JDK v1.8.0_60. +1 unit 2m 0s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. +1 unit 3m 41s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_60. -1 unit 0m 23s hadoop-yarn-api in the patch failed with JDK v1.7.0_79. +1 unit 2m 13s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. +1 unit 3m 52s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 45m 22s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields JDK v1.7.0_79 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772445/YARN-4234-2015-11-13.1.patch JIRA Issue YARN-4234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 94f484aac299 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / 855d529 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9693/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9693/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          TimelineClientImpl is doing two many things. Let's have a DirectTimelineWriter vs HDFSTimelineWriter which can encapsulate functionality.

          New patch addressed this comment

          Show
          xgong Xuan Gong added a comment - TimelineClientImpl is doing two many things. Let's have a DirectTimelineWriter vs HDFSTimelineWriter which can encapsulate functionality. New patch addressed this comment
          Hide
          xgong Xuan Gong added a comment -

          Thanks for the review.

          Do we want to add those configs into yarn-default?

          Probably no. Those are configurations only for ats_v_1.5.

          TimelineEntityGroupId, its string prefix does not contain the underscore, which appears to be inconsistent with app attempt id and application id. This may also be error pruning?

          It follows the pattern from containerId. The reason why its string prefix does not contain the underscore is because that we have fromString function.

          We may want to make it clear in the javadoc that some timeline client APIs are for ATS v1.5?

          DONE

          Show
          xgong Xuan Gong added a comment - Thanks for the review. Do we want to add those configs into yarn-default? Probably no. Those are configurations only for ats_v_1.5. TimelineEntityGroupId, its string prefix does not contain the underscore, which appears to be inconsistent with app attempt id and application id. This may also be error pruning? It follows the pattern from containerId. The reason why its string prefix does not contain the underscore is because that we have fromString function. We may want to make it clear in the javadoc that some timeline client APIs are for ATS v1.5? DONE
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi Xuan Gong, thanks for the update! General design LGTM, a few quick issues:

          1. Do we want to add those configs into yarn-default?
          2. TimelineEntityGroupId, its string prefix does not contain the underscore, which appears to be inconsistent with app attempt id and application id. This may also be error pruning?
          3. We may want to make it clear in the javadoc that some timeline client APIs are for ATS v1.5?
          4. For sanity check, I agree with Naganarasimha G R that our current approach requires the client config and server config to be consistent. I'm fine with moving forward here, but we may want to consider some mechanisms to let ATS v1 server identify its own version and configs in future.
          Show
          gtCarrera9 Li Lu added a comment - Hi Xuan Gong , thanks for the update! General design LGTM, a few quick issues: Do we want to add those configs into yarn-default? TimelineEntityGroupId, its string prefix does not contain the underscore, which appears to be inconsistent with app attempt id and application id. This may also be error pruning? We may want to make it clear in the javadoc that some timeline client APIs are for ATS v1.5? For sanity check, I agree with Naganarasimha G R that our current approach requires the client config and server config to be consistent. I'm fine with moving forward here, but we may want to consider some mechanisms to let ATS v1 server identify its own version and configs in future.
          Hide
          xgong Xuan Gong added a comment -

          renamed several configurations

          Show
          xgong Xuan Gong added a comment - renamed several configurations
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Li Lu (inactive) it would be helpful if you can also verify yarn.timeline-service.enabled is a client only config as explained in the documentation. From My side apart from RM's SystemMetricPublisher using it, i dont see it anyother code in the server using it.
          And any thoughts about the approach for getting the Version and related conf from the server side ?
          Also i have commented the same in YARN-4183.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Li Lu (inactive) it would be helpful if you can also verify yarn.timeline-service.enabled is a client only config as explained in the documentation. From My side apart from RM's SystemMetricPublisher using it, i dont see it anyother code in the server using it. And any thoughts about the approach for getting the Version and related conf from the server side ? Also i have commented the same in YARN-4183 .
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi Naganarasimha G R, I'm not extremely familiar with the delegation token stuff in ATS v1, but probably you'd like to mention your comments here in YARN-4183?

          Also, I'd like to double check if yarn.timeline-service.enabled is a client only config?

          Show
          gtCarrera9 Li Lu added a comment - Hi Naganarasimha G R , I'm not extremely familiar with the delegation token stuff in ATS v1, but probably you'd like to mention your comments here in YARN-4183 ? Also, I'd like to double check if yarn.timeline-service.enabled is a client only config?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Jason Lowe & Li Lu (inactive),
          AFAIK, YARN-4183 wanted to address lil more than that what is captured by Li Lu (inactive) (related to whether client should fail when timelineserver is not reachable....).
          But what i dont understand(or missing), as per the current code (ATSv1) is, only if the "yarn.timeline-service.enabled" is enabled only then the delegation tokens are created automatically in the YARNClientImpl. So not sure why Jonathan Eagles was pointing out all apps were trying to get delegation tokens in YARN-4183. I understand SystemMetric Publisher (Server) need not check for this configuration but not sure why client should depend on some server side configuration?
          Additionally we already have client side configuration "yarn.timeline-service.client.best-effort" (realized it now) if when configured will not fail the YARNClient.SubmitApplication if fail to get the delegation tokens.

          Hope My understanding about the problem is correct, If correct then as part of YARN-4183 we need to just remove the check for "yarn.timeline-service.enabled" being used in SystemMetricsPublisher, Please correct me if i am wrong.

          And coming to this patch(support for 1.5) i can envisage it as follows:

          • Server will be configured with "TIMELINE_SERVICE_VERSION" based on which appropriate timeline handler is selected
          • Client apps who ever want to communicate with Timeline server will enable "yarn.timeline-service.enabled"
          • If security and "yarn.timeline-service.enabled" are enabled then delegation token is got in the yarn client as earlier
          • when the timeline client is intitialized it contacts server for the version and related configs and once it receives it initializes itself.
          • If user tries to use invalid methods (not appropriate to the server timeline version) then timelineclient throws exception

          Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Jason Lowe & Li Lu (inactive) , AFAIK, YARN-4183 wanted to address lil more than that what is captured by Li Lu (inactive) (related to whether client should fail when timelineserver is not reachable....). But what i dont understand(or missing), as per the current code (ATSv1) is, only if the "yarn.timeline-service.enabled" is enabled only then the delegation tokens are created automatically in the YARNClientImpl. So not sure why Jonathan Eagles was pointing out all apps were trying to get delegation tokens in YARN-4183 . I understand SystemMetric Publisher (Server) need not check for this configuration but not sure why client should depend on some server side configuration? Additionally we already have client side configuration "yarn.timeline-service.client.best-effort" (realized it now) if when configured will not fail the YARNClient.SubmitApplication if fail to get the delegation tokens. Hope My understanding about the problem is correct, If correct then as part of YARN-4183 we need to just remove the check for "yarn.timeline-service.enabled" being used in SystemMetricsPublisher, Please correct me if i am wrong. And coming to this patch(support for 1.5) i can envisage it as follows: Server will be configured with "TIMELINE_SERVICE_VERSION" based on which appropriate timeline handler is selected Client apps who ever want to communicate with Timeline server will enable "yarn.timeline-service.enabled" If security and "yarn.timeline-service.enabled" are enabled then delegation token is got in the yarn client as earlier when the timeline client is intitialized it contacts server for the version and related configs and once it receives it initializes itself. If user tries to use invalid methods (not appropriate to the server timeline version) then timelineclient throws exception Thoughts ?
          Hide
          xgong Xuan Gong added a comment -

          Removed TimelineEntityGroupIdProto and TimelineEntityGroupIdPBImpl

          Show
          xgong Xuan Gong added a comment - Removed TimelineEntityGroupIdProto and TimelineEntityGroupIdPBImpl
          Hide
          gtCarrera9 Li Lu added a comment -

          I think here are some miscommunications, please feel free to correct me if I'm wrong. Here are actually two problems we're discussing:

          • For ATS v1.5 entity write patch, the client should not contact the server. In this way we're offloading the traffic from the centralized server, which became a significant problem of ATS v1.
          • For ATS compatibility issue raised in YARN-4183, we need to come up a mechanism to let the clients and server coordinate. Specifically, the client may want to know the server's ATS version or where exactly it needs to write the timeline entities. Client side configurations may not be consistent with the server in this case. So, maybe we want some mechanisms to coordinate on this?

          Not sure if I got the points right but to me those two points are independent?

          Show
          gtCarrera9 Li Lu added a comment - I think here are some miscommunications, please feel free to correct me if I'm wrong. Here are actually two problems we're discussing: For ATS v1.5 entity write patch, the client should not contact the server. In this way we're offloading the traffic from the centralized server, which became a significant problem of ATS v1. For ATS compatibility issue raised in YARN-4183 , we need to come up a mechanism to let the clients and server coordinate. Specifically, the client may want to know the server's ATS version or where exactly it needs to write the timeline entities. Client side configurations may not be consistent with the server in this case. So, maybe we want some mechanisms to coordinate on this? Not sure if I got the points right but to me those two points are independent?
          Hide
          jlowe Jason Lowe added a comment -

          One issue of having the client need to contact the server is it adds a dependency that ATS v1.5 is explicitly trying to remove. Today we are running jobs that work just fine whether the ATS is up or not. If the client needs to fetch anything from the ATS then jobs stop flowing as soon as the ATS is down. Since ATS v1 and v1.5 are not HA that's undesirable. It puts the ATS on the critical path for jobs flowing through the cluster.

          Show
          jlowe Jason Lowe added a comment - One issue of having the client need to contact the server is it adds a dependency that ATS v1.5 is explicitly trying to remove. Today we are running jobs that work just fine whether the ATS is up or not. If the client needs to fetch anything from the ATS then jobs stop flowing as soon as the ATS is down. Since ATS v1 and v1.5 are not HA that's undesirable. It puts the ATS on the critical path for jobs flowing through the cluster.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Naganarasimha G R, this is a good suggestion. Maybe we can put those self-descriptive features in a restful endpoint (I'm just thinking out loud)?

          Show
          gtCarrera9 Li Lu added a comment - Thanks Naganarasimha G R , this is a good suggestion. Maybe we can put those self-descriptive features in a restful endpoint (I'm just thinking out loud)?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Li Lu (inactive), for updating offline about the approach of ATS 1.5, it seems to be different than existing approaches in V1 and V2. Timeline client is able to write directly to HDFS without the knowledge of ATS Server(Reader).
          But one small suggestion, Can the configuration information(like activePath) related to 1.5 be got from server rather than client doing it based on its local configuration, this avoid configurations conflicts? hope my understanding is right. correct me if i am wrong.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Li Lu (inactive) , for updating offline about the approach of ATS 1.5, it seems to be different than existing approaches in V1 and V2. Timeline client is able to write directly to HDFS without the knowledge of ATS Server(Reader). But one small suggestion, Can the configuration information(like activePath) related to 1.5 be got from server rather than client doing it based on its local configuration, this avoid configurations conflicts? hope my understanding is right. correct me if i am wrong.
          Hide
          gtCarrera9 Li Lu added a comment -

          Well let me clarify one thing: ATS v1.5 supports all ATS v1 features, while ATS v2 is not compatible with ATS v1 and 1.5. Therefore, how about this:

          timeline-service.version: 1 -> only ATS v1 features are supported.
          timeline-service.version: 1.5 -> supports ATS v1 and v1.5 features.
          timeline-service.version: 2 -> only ATS v2 features are supported.

          This describes the feature set on the server side of ATS. The client can use this information from server to perform sanity check. I agree with Naganarasimha G R that checking this config on the client side does not quite make sense, and clearly we need some mechanisms to let the client know the current running version of ATS. However, I think this is orthogonal to the config key proposed in YARN-4183 (and added here).

          Show
          gtCarrera9 Li Lu added a comment - Well let me clarify one thing: ATS v1.5 supports all ATS v1 features, while ATS v2 is not compatible with ATS v1 and 1.5. Therefore, how about this: timeline-service.version: 1 -> only ATS v1 features are supported. timeline-service.version: 1.5 -> supports ATS v1 and v1.5 features. timeline-service.version: 2 -> only ATS v2 features are supported. This describes the feature set on the server side of ATS. The client can use this information from server to perform sanity check. I agree with Naganarasimha G R that checking this config on the client side does not quite make sense, and clearly we need some mechanisms to let the client know the current running version of ATS. However, I think this is orthogonal to the config key proposed in YARN-4183 (and added here).
          Hide
          gtCarrera9 Li Lu added a comment -

          Naganarasimha G R Sure, we can certainly add those extra sanity checks on the client. However, I believe anyways we need this config on the server side to describe the highest supported version number.

          Show
          gtCarrera9 Li Lu added a comment - Naganarasimha G R Sure, we can certainly add those extra sanity checks on the client. However, I believe anyways we need this config on the server side to describe the highest supported version number.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Li Lu (inactive), IIUC YARN-3488 is talking about if this server side configuration is configured differently in the client then though the user calls the right API's based on the client side ATS version, it will not be successfull in the server.
          So was wondering while starting of the timelineclient whether we can connect to the backend timeline server and check whether the server is for the specified ATS version in the client, or as YARN-3488 suggests receive the information related to the ATS version as part of RM register.Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Li Lu (inactive) , IIUC YARN-3488 is talking about if this server side configuration is configured differently in the client then though the user calls the right API's based on the client side ATS version, it will not be successfull in the server. So was wondering while starting of the timelineclient whether we can connect to the backend timeline server and check whether the server is for the specified ATS version in the client, or as YARN-3488 suggests receive the information related to the ATS version as part of RM register.Thoughts ?
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi Naganarasimha G R, IIUC the TIMELINE_SERVICE_VERSION config is describing the server's version. The client API implementation will perform sanity check to avoid calling a wrong (lower) version of API to the server. This is orthogonal to assuming to a particular version of ATS. The application can handle the logic by itself to decide which correct API to call. Am I missing anything here? Thanks.

          Show
          gtCarrera9 Li Lu added a comment - Hi Naganarasimha G R , IIUC the TIMELINE_SERVICE_VERSION config is describing the server's version. The client API implementation will perform sanity check to avoid calling a wrong (lower) version of API to the server. This is orthogonal to assuming to a particular version of ATS. The application can handle the logic by itself to decide which correct API to call. Am I missing anything here? Thanks.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Xuan Gong & Li Lu (inactive),
          Before we go ahead with the supporting clients mentioning of ATS version, it would be better to have a look at YARN-3488, which talks about issues of each application having client side configuration of which timeline version to use.
          Basically it would be better if server side validations happen, if client side then possibilities are more that app will assume to use a particular version of the ATS but the server can be having different timeline server running.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Xuan Gong & Li Lu (inactive) , Before we go ahead with the supporting clients mentioning of ATS version, it would be better to have a look at YARN-3488 , which talks about issues of each application having client side configuration of which timeline version to use. Basically it would be better if server side validations happen, if client side then possibilities are more that app will assume to use a particular version of the ATS but the server can be having different timeline server running.
          Hide
          xgong Xuan Gong added a comment -

          Attached a new patch with the renamed configurations.

          CacheId -> TimelineEntityGroupId ? And similarly rename it everywhere.

          DONE

          Also move this to org.apache.hadoop.yarn.api.records.timeline.

          DONE

          Move CacheIdProto from yarn_protos.proto also accordingly

          I can not find a better place. Just keep it in yarn_protos.proto.

          Reorder the API parameters

          DONE.

          entity-file-fd.flush-interval-secs -> entity-file-store.fd-flush-interval-secs

          DONE

          Similarly entity-file-fd.clean-interval-sec and entity-file-fd.retain-secs

          DONE

          Why do we need TIMELINE_SERVICE_PLUGIN_ENABLED especially if we also have this TimelineEntityGroupId/CacheId as part of the writer API?

          REMOVED

          TimelineClientImpl is doing two many things. Let's have a DirectTimelineWriter vs HDFSTimelineWriter which can encapsulate functionality.

          Will do it later.

          Also, I add a new configuration: TIMELINE_SERVICE_VERSION. So, if we want to use ats 1.5, we need set 1.5 for this configuration. And when we try to use new api for ats1.5, a sanity check would be enforced.

          Show
          xgong Xuan Gong added a comment - Attached a new patch with the renamed configurations. CacheId -> TimelineEntityGroupId ? And similarly rename it everywhere. DONE Also move this to org.apache.hadoop.yarn.api.records.timeline. DONE Move CacheIdProto from yarn_protos.proto also accordingly I can not find a better place. Just keep it in yarn_protos.proto. Reorder the API parameters DONE. entity-file-fd.flush-interval-secs -> entity-file-store.fd-flush-interval-secs DONE Similarly entity-file-fd.clean-interval-sec and entity-file-fd.retain-secs DONE Why do we need TIMELINE_SERVICE_PLUGIN_ENABLED especially if we also have this TimelineEntityGroupId/CacheId as part of the writer API? REMOVED TimelineClientImpl is doing two many things. Let's have a DirectTimelineWriter vs HDFSTimelineWriter which can encapsulate functionality. Will do it later. Also, I add a new configuration: TIMELINE_SERVICE_VERSION. So, if we want to use ats 1.5, we need set 1.5 for this configuration. And when we try to use new api for ats1.5, a sanity check would be enforced.
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Please also look at https://issues.apache.org/jira/browse/YARN-4183?focusedCommentId=14999670&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14999670 for more thoughts on the API.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          I have some naming and source-code comments that I just cannot hold back on

          • CacheId -> TimelineEntityGroupId ? And similarly rename it everywhere. I just cannot think of a top-level YARN CacheId
          • Also move this to org.apache.hadoop.yarn.api.records.timeline.
          • Move CacheIdProto from yarn_protos.proto also accordingly
          • Reorder the API parameters
            putEntities(ApplicationAttemptId appAttemptId, TimelineEntityGroupId groupId, TimelineEntity... entities)
            
          • Why do we need TIMELINE_SERVICE_PLUGIN_ENABLED especially if we also have this TimelineEntityGroupId/CacheId as part of the writer API? We should just remove this if it is not needed.
          • entity-file-fd.flush-interval-secs -> entity-file-store.fd-flush-interval-secs
          • Similarly entity-file-fd.clean-interval-sec and entity-file-fd.retain-secs
          • TimelineClientImpl is doing two many things. Let's have a DirectTimelineWriter vs HDFSTimelineWriter which can encapsulate functionality.
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - I have some naming and source-code comments that I just cannot hold back on CacheId -> TimelineEntityGroupId ? And similarly rename it everywhere. I just cannot think of a top-level YARN CacheId Also move this to org.apache.hadoop.yarn.api.records.timeline. Move CacheIdProto from yarn_protos.proto also accordingly Reorder the API parameters putEntities(ApplicationAttemptId appAttemptId, TimelineEntityGroupId groupId, TimelineEntity... entities) Why do we need TIMELINE_SERVICE_PLUGIN_ENABLED especially if we also have this TimelineEntityGroupId/CacheId as part of the writer API? We should just remove this if it is not needed. entity-file-fd.flush-interval-secs -> entity-file-store.fd-flush-interval-secs Similarly entity-file-fd.clean-interval-sec and entity-file-fd.retain-secs TimelineClientImpl is doing two many things. Let's have a DirectTimelineWriter vs HDFSTimelineWriter which can encapsulate functionality.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 3m 12s trunk passed
          +1 compile 0m 47s trunk passed with JDK v1.8.0_60
          +1 compile 0m 48s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 27s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 27s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 48s the patch passed with JDK v1.8.0_60
          +1 cc 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          +1 compile 0m 48s the patch passed with JDK v1.7.0_79
          +1 cc 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -1 checkstyle 0m 27s Patch generated 30 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 264).
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 29s the patch passed with JDK v1.7.0_79
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_60.
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_79.
          +1 unit 2m 7s hadoop-yarn-common in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 19s Patch generated 3 ASF License warnings.
          30m 8s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771595/YARN-4234.20151110.1.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc
          uname Linux 2cd3a73bf443 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 73b94d7
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9649/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9649/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9649/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9649/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9649/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 12s trunk passed +1 compile 0m 47s trunk passed with JDK v1.8.0_60 +1 compile 0m 48s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 27s trunk passed +1 mvneclipse 0m 26s trunk passed -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 7s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 27s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 54s the patch passed +1 compile 0m 48s the patch passed with JDK v1.8.0_60 +1 cc 0m 48s the patch passed +1 javac 0m 48s the patch passed +1 compile 0m 48s the patch passed with JDK v1.7.0_79 +1 cc 0m 48s the patch passed +1 javac 0m 48s the patch passed -1 checkstyle 0m 27s Patch generated 30 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 264). +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 52s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 29s the patch passed with JDK v1.7.0_79 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_60. +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_60. +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_79. +1 unit 2m 7s hadoop-yarn-common in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 3 ASF License warnings. 30m 8s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771595/YARN-4234.20151110.1.patch JIRA Issue YARN-4234 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc uname Linux 2cd3a73bf443 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 73b94d7 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9649/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9649/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9649/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9649/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9649/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          Fix findbugs warning

          Show
          xgong Xuan Gong added a comment - Fix findbugs warning
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 3m 4s trunk passed
          +1 compile 0m 49s trunk passed with JDK v1.8.0_66
          +1 compile 0m 48s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 26s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 3s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 14s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 49s the patch passed with JDK v1.8.0_66
          +1 cc 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_85
          +1 cc 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -1 checkstyle 0m 26s Patch generated 30 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 264).
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api introduced 1 new FindBugs issues.
          -1 findbugs 1m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common introduced 4 new FindBugs issues.
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 20s the patch passed with JDK v1.7.0_85
          +1 unit 0m 19s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          -1 asflicense 0m 18s Patch generated 1 ASF License warnings.
          29m 8s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
            org.apache.hadoop.yarn.api.records.CacheId.fromString(String) concatenates strings using + in a loop At CacheId.java:in a loop At CacheId.java:[line 116]
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
            logFD could be null and is guaranteed to be dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) Dereferenced at TimelineClientImpl.java:is guaranteed to be dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) Dereferenced at TimelineClientImpl.java:[line 1081]
            Nullcheck of logFD at line 1072 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) At TimelineClientImpl.java:1072 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) At TimelineClientImpl.java:[line 1072]
            Nullcheck of logFD at line 1127 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.closeFDs(Map, boolean) At TimelineClientImpl.java:1127 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.closeFDs(Map, boolean) At TimelineClientImpl.java:[line 1127]
            Nullcheck of logFD at line 1021 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.flushFDMap(Map, boolean) At TimelineClientImpl.java:1021 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.flushFDMap(Map, boolean) At TimelineClientImpl.java:[line 1021]



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771509/YARN-4234.20151109.patch
          JIRA Issue YARN-4234
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc
          uname Linux 510497a16719 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 0eb9c60
          Default Java 1.7.0_85
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_85
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9648/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9648/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 4s trunk passed +1 compile 0m 49s trunk passed with JDK v1.8.0_66 +1 compile 0m 48s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 26s trunk passed +1 mvneclipse 0m 24s trunk passed -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 3s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 14s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 49s the patch passed +1 compile 0m 49s the patch passed with JDK v1.8.0_66 +1 cc 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_85 +1 cc 0m 47s the patch passed +1 javac 0m 47s the patch passed -1 checkstyle 0m 26s Patch generated 30 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 235, now 264). +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api introduced 1 new FindBugs issues. -1 findbugs 1m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common introduced 4 new FindBugs issues. +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 20s the patch passed with JDK v1.7.0_85 +1 unit 0m 19s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 18s Patch generated 1 ASF License warnings. 29m 8s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api   org.apache.hadoop.yarn.api.records.CacheId.fromString(String) concatenates strings using + in a loop At CacheId.java:in a loop At CacheId.java: [line 116] FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common   logFD could be null and is guaranteed to be dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) Dereferenced at TimelineClientImpl.java:is guaranteed to be dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) Dereferenced at TimelineClientImpl.java: [line 1081]   Nullcheck of logFD at line 1072 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) At TimelineClientImpl.java:1072 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.cleanInActiveFDsforMap(Map, long, boolean) At TimelineClientImpl.java: [line 1072]   Nullcheck of logFD at line 1127 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.closeFDs(Map, boolean) At TimelineClientImpl.java:1127 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.closeFDs(Map, boolean) At TimelineClientImpl.java: [line 1127]   Nullcheck of logFD at line 1021 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.flushFDMap(Map, boolean) At TimelineClientImpl.java:1021 of value previously dereferenced in org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.flushFDMap(Map, boolean) At TimelineClientImpl.java: [line 1021] Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-10 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771509/YARN-4234.20151109.patch JIRA Issue YARN-4234 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc uname Linux 510497a16719 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 0eb9c60 Default Java 1.7.0_85 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_85 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9648/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9648/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9648/console This message was automatically generated.

            People

            • Assignee:
              xgong Xuan Gong
              Reporter:
              xgong Xuan Gong
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development