Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4233 YARN Timeline Service plugin: ATS v1.5
  3. YARN-4265

Provide new timeline plugin storage to support fine-grained entity caching

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: timelineserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To support the newly proposed APIs in YARN-4234, we need to create a new plugin timeline store. The store may have similar behavior as the EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id granularity, instead of application id granularity. Let's have this storage as a standalone one, instead of updating EntityFileTimelineStore, to keep the existing store (EntityFileTimelineStore) stable.

      1. YARN-4265-trunk.008.patch
        114 kB
        Li Lu
      2. YARN-4265-trunk.007.patch
        114 kB
        Li Lu
      3. YARN-4265-trunk.006.patch
        112 kB
        Li Lu
      4. YARN-4265-trunk.005.patch
        112 kB
        Li Lu
      5. YARN-4265-trunk.004.patch
        107 kB
        Li Lu
      6. YARN-4265-trunk.003.patch
        104 kB
        Li Lu
      7. YARN-4265-trunk.002.patch
        104 kB
        Li Lu
      8. YARN-4265-trunk.001.patch
        106 kB
        Li Lu
      9. YARN-4265-branch-2-addendum.patch
        1.0 kB
        Rohith Sharma K S
      10. YARN-4265.YARN-4234.002.patch
        105 kB
        Li Lu
      11. YARN-4265.YARN-4234.001.patch
        104 kB
        Li Lu
      12. YANR-4265-branch-2.8-addendum.patch
        1.0 kB
        Rohith Sharma K S

        Issue Links

          Activity

          Hide
          gtCarrera9 Li Lu added a comment -

          We need to use the finalized APIs in YARN-4234 to build the reading code routine for ATS 1.5 storage.

          Show
          gtCarrera9 Li Lu added a comment - We need to use the finalized APIs in YARN-4234 to build the reading code routine for ATS 1.5 storage.
          Hide
          gtCarrera9 Li Lu added a comment -

          This is a proof-of-concept patch for the proposed improvement. This patch is largely based on the work in YARN-3942. In this patch we added a new storage plugin to accommodate the newly added cache id concept. As in YARN-4234, entity logs are stored in <active/done dir name>/app_id/attampt_id/<prefix>-cache_id.log. The newly added plugin can with with this new directory structure, and can refresh (reload) entity logs in a cache id granularity (rather than application granularity).

          Show
          gtCarrera9 Li Lu added a comment - This is a proof-of-concept patch for the proposed improvement. This patch is largely based on the work in YARN-3942 . In this patch we added a new storage plugin to accommodate the newly added cache id concept. As in YARN-4234 , entity logs are stored in <active/done dir name>/app_id/attampt_id/<prefix>-cache_id.log. The newly added plugin can with with this new directory structure, and can refresh (reload) entity logs in a cache id granularity (rather than application granularity).
          Hide
          gtCarrera9 Li Lu added a comment -

          Another key addition in this patch, compare to YARN-3942, is a "cache id plugin" mechanism. One general problem the plugin needs to solve is "which exact storage should I look at on a read request like ...". The user can add a plugin (for example, a Tez plugin) in the configuration. At run time, on each read request, the plugin can help deciding if we need to look at the summary storage only or the detailed (entity) storage. In this way, the whole storage decision process of ATS 1.5 fine-grain cache storage will be:

          • On writes, the application can decide if an entity should go to the summary storage or the detailed storage. If it should go to the latter, the application need to provide a cache id for this entity.
          • On reads, the application provides a plugin to decide if the look up can be performed on the summery storage, or on the detailed storage. If it's the latter, the plugin provides a cache id so that the storage should look at.
          Show
          gtCarrera9 Li Lu added a comment - Another key addition in this patch, compare to YARN-3942 , is a "cache id plugin" mechanism. One general problem the plugin needs to solve is "which exact storage should I look at on a read request like ...". The user can add a plugin (for example, a Tez plugin) in the configuration. At run time, on each read request, the plugin can help deciding if we need to look at the summary storage only or the detailed (entity) storage. In this way, the whole storage decision process of ATS 1.5 fine-grain cache storage will be: On writes, the application can decide if an entity should go to the summary storage or the detailed storage. If it should go to the latter, the application need to provide a cache id for this entity. On reads, the application provides a plugin to decide if the look up can be performed on the summery storage, or on the detailed storage. If it's the latter, the plugin provides a cache id so that the storage should look at.
          Hide
          gtCarrera9 Li Lu added a comment -

          This patch relies on YARN-3942.002 and YARN-4234.3 patches.

          Show
          gtCarrera9 Li Lu added a comment - This patch relies on YARN-3942 .002 and YARN-4234 .3 patches.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Li Lu! This looks like most of the patch is a copy of the entity timeline store from YARN-3942 with a few edits, so I'm sorta reviewing my own code here. As such I did a diff of the patch from this JIRA and the one from YARN-3942 so I could focus on what's changed. I'll defer to others to review the parts that are identical to YARN-3942. Eventually I can see this being a superset of YARN-3942, since it can cache to memory and either cache everything or a subset based on what the plugins decide.

          TIMELINE_SERVICE_PLUGIN_ENABLED and DEFAULT_TIMELINE_SERVICE_PLUGIN_ENABLED are not needed.

          Is there a reason we're not using the Configuration.getInstances method or the ReflectionUtils methods to handle plugin loading?

          If no plugins are configured (which is the default behavior), do we want a fallback plugin that emulates what YARN-3942 is doing?

          Are there plans to support the leveldb store as an alternative to the memory store for the detail timeline store? There was concern that a single dag could overwhelm the server, and storing it to leveldb instead of the memory store would be one way to try to mitigate that. I'm wondering if the class to use for the detail log timeline store should be configurable with MemoryTimelineStore as the default. Could also do this as a followup JIRA if necessary.

          Should add entries to yarn-default.xml for the new properties?

          Do we want to log at the info level that a path is being skipped during the scan? The store can end up scanning fairly often in practice, so this could end up logging a lot for just one path per scan. I'm wondering if making it a debug log is more appropriate.

          Comment in getDoneAppPath mentions a cache ID but it's using an app ID.

          Nit: Indentation is off at the start of the YarnConfiguration patch hunk.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Li Lu ! This looks like most of the patch is a copy of the entity timeline store from YARN-3942 with a few edits, so I'm sorta reviewing my own code here. As such I did a diff of the patch from this JIRA and the one from YARN-3942 so I could focus on what's changed. I'll defer to others to review the parts that are identical to YARN-3942 . Eventually I can see this being a superset of YARN-3942 , since it can cache to memory and either cache everything or a subset based on what the plugins decide. TIMELINE_SERVICE_PLUGIN_ENABLED and DEFAULT_TIMELINE_SERVICE_PLUGIN_ENABLED are not needed. Is there a reason we're not using the Configuration.getInstances method or the ReflectionUtils methods to handle plugin loading? If no plugins are configured (which is the default behavior), do we want a fallback plugin that emulates what YARN-3942 is doing? Are there plans to support the leveldb store as an alternative to the memory store for the detail timeline store? There was concern that a single dag could overwhelm the server, and storing it to leveldb instead of the memory store would be one way to try to mitigate that. I'm wondering if the class to use for the detail log timeline store should be configurable with MemoryTimelineStore as the default. Could also do this as a followup JIRA if necessary. Should add entries to yarn-default.xml for the new properties? Do we want to log at the info level that a path is being skipped during the scan? The store can end up scanning fairly often in practice, so this could end up logging a lot for just one path per scan. I'm wondering if making it a debug log is more appropriate. Comment in getDoneAppPath mentions a cache ID but it's using an app ID. Nit: Indentation is off at the start of the YarnConfiguration patch hunk.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Jason Lowe! Yes the current POC version is heavily based on YARN-3942. I'm doing some refactoring right now but the general logic should be pretty similar to your logic (thank for the work! ). I'll address your comments in the next version of the patch.

          If no plugins are configured (which is the default behavior), do we want a fallback plugin that emulates what YARN-3942 is doing?

          That's a nice suggestion. We can build a plugin that simply return the appid to simulate this.

          Are there plans to support the leveldb store as an alternative to the memory store for the detail timeline store? There was concern that a single dag could overwhelm the server, and storing it to leveldb instead of the memory store would be one way to try to mitigate that. I'm wondering if the class to use for the detail log timeline store should be configurable with MemoryTimelineStore as the default. Could also do this as a followup JIRA if necessary.

          Will do.

          Should add entries to yarn-default.xml for the new properties?

          Yes. Will add them.

          Do we want to log at the info level that a path is being skipped during the scan? The store can end up scanning fairly often in practice, so this could end up logging a lot for just one path per scan. I'm wondering if making it a debug log is more appropriate.

          Sure.

          Show
          gtCarrera9 Li Lu added a comment - Thanks Jason Lowe ! Yes the current POC version is heavily based on YARN-3942 . I'm doing some refactoring right now but the general logic should be pretty similar to your logic (thank for the work! ). I'll address your comments in the next version of the patch. If no plugins are configured (which is the default behavior), do we want a fallback plugin that emulates what YARN-3942 is doing? That's a nice suggestion. We can build a plugin that simply return the appid to simulate this. Are there plans to support the leveldb store as an alternative to the memory store for the detail timeline store? There was concern that a single dag could overwhelm the server, and storing it to leveldb instead of the memory store would be one way to try to mitigate that. I'm wondering if the class to use for the detail log timeline store should be configurable with MemoryTimelineStore as the default. Could also do this as a followup JIRA if necessary. Will do. Should add entries to yarn-default.xml for the new properties? Yes. Will add them. Do we want to log at the info level that a path is being skipped during the scan? The store can end up scanning fairly often in practice, so this could end up logging a lot for just one path per scan. I'm wondering if making it a debug log is more appropriate. Sure.
          Hide
          gtCarrera9 Li Lu added a comment -

          I refactored my code and added some unit tests. The current patch only depends on YARN-4234. I addressed comments raised by Jason Lowe in the previous round of review, with two pending actions that I think we can address in separate JIRAs:

          1. Build a "fall-back" plugin to have the same behavior of YARN-3942.
          2. Make caching storage pluggable.

          For the second item, right now my patch does not depend on YARN-4219. So I'm setting the v1.5 plugin storage to use the memory storage system as the "caching" storage, similar to YARN-3942.

          I'm adding the v1.5 plugin storage (EntityGroupFSTimelineStore) in a module called hadoop-yarn-server-timeline-pluginstorage. This is slightly different to YARN-3942. We need a separate module because the new v1.5 storage depends on yarn-client, but we don't want ats server depends on yarn-client (The v1.5 storage is not a purely server side storage.). I'm naming it as "pluginstorage" because I'm considering to put the leveldb caching storage into this extension as well.

          Show
          gtCarrera9 Li Lu added a comment - I refactored my code and added some unit tests. The current patch only depends on YARN-4234 . I addressed comments raised by Jason Lowe in the previous round of review, with two pending actions that I think we can address in separate JIRAs: Build a "fall-back" plugin to have the same behavior of YARN-3942 . Make caching storage pluggable. For the second item, right now my patch does not depend on YARN-4219 . So I'm setting the v1.5 plugin storage to use the memory storage system as the "caching" storage, similar to YARN-3942 . I'm adding the v1.5 plugin storage (EntityGroupFSTimelineStore) in a module called hadoop-yarn-server-timeline-pluginstorage. This is slightly different to YARN-3942 . We need a separate module because the new v1.5 storage depends on yarn-client, but we don't want ats server depends on yarn-client (The v1.5 storage is not a purely server side storage.). I'm naming it as "pluginstorage" because I'm considering to put the leveldb caching storage into this extension as well.
          Hide
          gtCarrera9 Li Lu added a comment -

          Update a new patch to fix some log issues and implement the getTimeline API.

          Show
          gtCarrera9 Li Lu added a comment - Update a new patch to fix some log issues and implement the getTimeline API.
          Hide
          djp Junping Du added a comment -

          I just commit YARN-4234. Li Lu, would you rebase your patch on latest trunk branch? Thanks!
          Hi Jason Lowe, I saw your comments above: "This looks like most of the patch is a copy of the entity timeline store from YARN-3942 with a few edits, so I'm sorta reviewing my own code here. As such I did a diff of the patch from this JIRA and the one from YARN-3942 so I could focus on what's changed. I'll defer to others to review the parts that are identical to YARN-3942. Eventually I can see this being a superset of YARN-3942, since it can cache to memory and either cache everything or a subset based on what the plugins decide." Are you OK with continue the review effort going with this patch? Or you have some other preferences?

          Show
          djp Junping Du added a comment - I just commit YARN-4234 . Li Lu , would you rebase your patch on latest trunk branch? Thanks! Hi Jason Lowe , I saw your comments above: "This looks like most of the patch is a copy of the entity timeline store from YARN-3942 with a few edits, so I'm sorta reviewing my own code here. As such I did a diff of the patch from this JIRA and the one from YARN-3942 so I could focus on what's changed. I'll defer to others to review the parts that are identical to YARN-3942 . Eventually I can see this being a superset of YARN-3942 , since it can cache to memory and either cache everything or a subset based on what the plugins decide." Are you OK with continue the review effort going with this patch? Or you have some other preferences?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 5s YARN-4265 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776931/YARN-4265.YARN-4234.002.patch
          JIRA Issue YARN-4265
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10080/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 5s YARN-4265 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776931/YARN-4265.YARN-4234.002.patch JIRA Issue YARN-4265 Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10080/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Junping Du! I just rebased my patch to the latest trunk.

          Show
          gtCarrera9 Li Lu added a comment - Thanks Junping Du ! I just rebased my patch to the latest trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 8m 11s trunk passed
          +1 compile 2m 2s trunk passed with JDK v1.8.0_66
          +1 compile 2m 15s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 2m 43s trunk passed
          +1 mvneclipse 0m 45s trunk passed
          -1 findbugs 3m 32s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          +1 javadoc 2m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 45s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 24s the patch passed
          +1 compile 2m 4s the patch passed with JDK v1.8.0_66
          +1 javac 2m 4s the patch passed
          +1 compile 2m 21s the patch passed with JDK v1.7.0_91
          +1 javac 2m 21s the patch passed
          -1 checkstyle 0m 29s Patch generated 54 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 345).
          +1 mvnsite 2m 47s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 3m 42s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          +1 javadoc 2m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 46s the patch passed with JDK v1.7.0_91
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 79m 12s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 19s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 81m 40s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 34s Patch generated 1 ASF License warnings.
          222m 18s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRM



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779316/YARN-4265-trunk.001.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 597a5836ed64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bb5df27
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10085/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10085/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 8m 11s trunk passed +1 compile 2m 2s trunk passed with JDK v1.8.0_66 +1 compile 2m 15s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 2m 43s trunk passed +1 mvneclipse 0m 45s trunk passed -1 findbugs 3m 32s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) +1 javadoc 2m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 45s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 24s the patch passed +1 compile 2m 4s the patch passed with JDK v1.8.0_66 +1 javac 2m 4s the patch passed +1 compile 2m 21s the patch passed with JDK v1.7.0_91 +1 javac 2m 21s the patch passed -1 checkstyle 0m 29s Patch generated 54 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 345). +1 mvnsite 2m 47s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 3m 42s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) +1 javadoc 2m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 46s the patch passed with JDK v1.7.0_91 +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 79m 12s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 19s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 81m 40s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 34s Patch generated 1 ASF License warnings. 222m 18s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRM Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779316/YARN-4265-trunk.001.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 597a5836ed64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bb5df27 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10085/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10085/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10085/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          BTW, I've got some offline discussion with Jason Lowe. In general, we agreed to move forward with only YARN-4265, which is built on top of YARN-4234. If I've got any misunderstanding, please feel free to let me know, Jason. Thanks!

          Show
          gtCarrera9 Li Lu added a comment - BTW, I've got some offline discussion with Jason Lowe . In general, we agreed to move forward with only YARN-4265 , which is built on top of YARN-4234 . If I've got any misunderstanding, please feel free to let me know, Jason. Thanks!
          Hide
          gtCarrera9 Li Lu added a comment -

          Sorry by saying YARN-4234 I'm actually meaning YARN-3942.

          Show
          gtCarrera9 Li Lu added a comment - Sorry by saying YARN-4234 I'm actually meaning YARN-3942 .
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu for clarification. Assume Jason is fine with going on with this patch, I quickly go through the v3 patch.
          with following comments so far (I haven't finished my review yet as patch is pretty big):

          In YarnConfiguration.java,

          TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_SCAN_INTERVAL_SECONDS_DEFAULT = 60;
          

          I noticed that we are setting 1 minutes as default scan interval but original patch in HDFS-3942 is 5 minutes. Why shall we do any update here? The same question on "app-cache-size", the default value in HDFS-3942 is 5 but here is 10. Any reason to update the value?

          In yarn-default.xml,

          +    <description>DFS path to store active application’s timeline data</description>
          ...
          +    <description>DFS path to store done application’s timeline data</description>
          

          DFS is very old name, use HDFS instead to be more clear.

          Why we don't have any default value specified in property of "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes"?

          In hadoop-yarn-server-timeline-pluginstorage/pom.xml,

          For EmptyTimelineEntityGroupPlugin.java, why we need this class? I didn't see any help provided even in tests. We should remove it if useless.

          In EntityCacheItem.java,
          We should have a description for this class in Javadoc.

          Can we optimize the synchronization logic here? Like in synchronized method refreshCache, we are intialize/start/stop TimelineDataManager (and MemoryTimelineStore) which is quite expensive and unnecessary to block other synchronized operations. Shall we move these operations out of synchronized block?

          +      LOG.warn("Error closing datamanager", e);
          

          I think we are closing store here instead of datamanager. Isn't it?

          +  public boolean needRefresh() {
          +    //TODO: make a config for cache freshness
          +    return (Time.monotonicNow() - lastRefresh > 10000);
          +  }
          

          Does refresh interval here need to do any coordination with scan interval specificed in "yarn.timeline-service.entity-group-fs-store.scan-interval-seconds"?

          In EntityGroupFSTimelineStore.java,

          +      if (appState != AppState.UNKNOWN) {
          +        appLogs = new AppLogs(applicationId, appDirPath, appState);
          +        LOG.debug("Create and try to add new appLogs to appIdLogMap for {}",
          +            applicationId);
          +        AppLogs oldAppLogs = appIdLogMap.putIfAbsent(applicationId, appLogs);
          +        if (oldAppLogs != null) {
          +          appLogs = oldAppLogs;
          +        }
          +      }
          

          This logic is very similiar with method of getAndSetActiveLog(). Can we consolidate them together?

          If parseSummaryLogs() is synchronized, it seems getSummaryLogs() should be synchronized too or the getter will get stale(half-done) result.

          Still checking if other multi-threads issues. More comments will come soon.

          Show
          djp Junping Du added a comment - Thanks Li Lu for clarification. Assume Jason is fine with going on with this patch, I quickly go through the v3 patch. with following comments so far (I haven't finished my review yet as patch is pretty big): In YarnConfiguration.java, TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_SCAN_INTERVAL_SECONDS_DEFAULT = 60; I noticed that we are setting 1 minutes as default scan interval but original patch in HDFS-3942 is 5 minutes. Why shall we do any update here? The same question on "app-cache-size", the default value in HDFS-3942 is 5 but here is 10. Any reason to update the value? In yarn-default.xml, + <description>DFS path to store active application’s timeline data</description> ... + <description>DFS path to store done application’s timeline data</description> DFS is very old name, use HDFS instead to be more clear. Why we don't have any default value specified in property of "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes"? In hadoop-yarn-server-timeline-pluginstorage/pom.xml, For EmptyTimelineEntityGroupPlugin.java, why we need this class? I didn't see any help provided even in tests. We should remove it if useless. In EntityCacheItem.java, We should have a description for this class in Javadoc. Can we optimize the synchronization logic here? Like in synchronized method refreshCache, we are intialize/start/stop TimelineDataManager (and MemoryTimelineStore) which is quite expensive and unnecessary to block other synchronized operations. Shall we move these operations out of synchronized block? + LOG.warn( "Error closing datamanager" , e); I think we are closing store here instead of datamanager. Isn't it? + public boolean needRefresh() { + //TODO: make a config for cache freshness + return (Time.monotonicNow() - lastRefresh > 10000); + } Does refresh interval here need to do any coordination with scan interval specificed in "yarn.timeline-service.entity-group-fs-store.scan-interval-seconds"? In EntityGroupFSTimelineStore.java, + if (appState != AppState.UNKNOWN) { + appLogs = new AppLogs(applicationId, appDirPath, appState); + LOG.debug( "Create and try to add new appLogs to appIdLogMap for {}" , + applicationId); + AppLogs oldAppLogs = appIdLogMap.putIfAbsent(applicationId, appLogs); + if (oldAppLogs != null ) { + appLogs = oldAppLogs; + } + } This logic is very similiar with method of getAndSetActiveLog(). Can we consolidate them together? If parseSummaryLogs() is synchronized, it seems getSummaryLogs() should be synchronized too or the getter will get stale(half-done) result. Still checking if other multi-threads issues. More comments will come soon.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Junping Du for the review! In the 002 patch I addressed most of the checkstyle problems, as well as most existing comments. Please feel free to add more. Some comments:

          I noticed that we are setting 1 minutes as default scan interval but original patch in HDFS-3942 is 5 minutes. Why shall we do any update here?

          For now I increased the default frequency to scan HDFS and pull timeline data. Having a 5-minute time interval means users are less likely to see any running status for apps that finish within 5 minutes. Right now I'm setting this value to 1 minute to reduce reader react time.

          The same question on "app-cache-size", the default value in HDFS-3942 is 5 but here is 10. Any reason to update the value?

          In YARN-3942, caching is performed on application level. In this patch, caching is performed in entity groups. Each application may have a few to tens of entity groups. Normally, there are slightly more active entity groups than active applications in the system. For now, I'm increasing this default value to hold slightly more entity groups in cache.

          Why we don't have any default value specified in property of "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes"?

          Plugins are provided by third-party applications such as Tez. Right now we cannot assume which exact entity group plugin the user is using, therefore we have to conservatively leave this config as empty.

          For EmptyTimelineEntityGroupPlugin.java, why we need this class? I didn't see any help provided even in tests. We should remove it if useless.

          Ah, nice catch. Removed it.

          Can we optimize the synchronization logic here? Like in synchronized method refreshCache, we are intialize/start/stop TimelineDataManager (and MemoryTimelineStore) which is quite expensive and unnecessary to block other synchronized operations. Shall we move these operations out of synchronized block?

          It's certainly doable. Right now I have yet to optimize this part because it's a little bit tricky to fine tune synchronization performance before we have a relatively stable starting point. Also, we're using fine-grained locking for each cached item in the reader cache, and cache refresh only happens infrequently (~10 secs by default), so maybe we'd like to stabilize the whole synchronization story before fine tune this part?

          Show
          gtCarrera9 Li Lu added a comment - Thanks Junping Du for the review! In the 002 patch I addressed most of the checkstyle problems, as well as most existing comments. Please feel free to add more. Some comments: I noticed that we are setting 1 minutes as default scan interval but original patch in HDFS-3942 is 5 minutes. Why shall we do any update here? For now I increased the default frequency to scan HDFS and pull timeline data. Having a 5-minute time interval means users are less likely to see any running status for apps that finish within 5 minutes. Right now I'm setting this value to 1 minute to reduce reader react time. The same question on "app-cache-size", the default value in HDFS-3942 is 5 but here is 10. Any reason to update the value? In YARN-3942 , caching is performed on application level. In this patch, caching is performed in entity groups. Each application may have a few to tens of entity groups. Normally, there are slightly more active entity groups than active applications in the system. For now, I'm increasing this default value to hold slightly more entity groups in cache. Why we don't have any default value specified in property of "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes"? Plugins are provided by third-party applications such as Tez. Right now we cannot assume which exact entity group plugin the user is using, therefore we have to conservatively leave this config as empty. For EmptyTimelineEntityGroupPlugin.java, why we need this class? I didn't see any help provided even in tests. We should remove it if useless. Ah, nice catch. Removed it. Can we optimize the synchronization logic here? Like in synchronized method refreshCache, we are intialize/start/stop TimelineDataManager (and MemoryTimelineStore) which is quite expensive and unnecessary to block other synchronized operations. Shall we move these operations out of synchronized block? It's certainly doable. Right now I have yet to optimize this part because it's a little bit tricky to fine tune synchronization performance before we have a relatively stable starting point. Also, we're using fine-grained locking for each cached item in the reader cache, and cache refresh only happens infrequently (~10 secs by default), so maybe we'd like to stabilize the whole synchronization story before fine tune this part?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 7m 36s trunk passed
          +1 compile 1m 45s trunk passed with JDK v1.8.0_66
          +1 compile 2m 4s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 2m 28s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          -1 findbugs 3m 19s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          +1 javadoc 2m 0s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 26s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 9s the patch passed
          +1 compile 1m 48s the patch passed with JDK v1.8.0_66
          +1 javac 1m 48s the patch passed
          +1 compile 2m 10s the patch passed with JDK v1.7.0_91
          +1 javac 2m 10s the patch passed
          -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300).
          +1 mvnsite 2m 30s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 1s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 3m 25s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 3m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 32 new issues (was 544, now 544).
          +1 javadoc 1m 51s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 23s the patch passed with JDK v1.7.0_91
          +1 unit 0m 19s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 56s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 79m 35s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 10s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 80m 43s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          226m 56s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780433/YARN-4265-trunk.002.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 14719e8d400e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 778146e
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10149/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10149/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 7m 36s trunk passed +1 compile 1m 45s trunk passed with JDK v1.8.0_66 +1 compile 2m 4s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 44s trunk passed -1 findbugs 3m 19s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) +1 javadoc 2m 0s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 26s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 9s the patch passed +1 compile 1m 48s the patch passed with JDK v1.8.0_66 +1 javac 1m 48s the patch passed +1 compile 2m 10s the patch passed with JDK v1.7.0_91 +1 javac 2m 10s the patch passed -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300). +1 mvnsite 2m 30s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 1s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 3m 25s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 3m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 32 new issues (was 544, now 544). +1 javadoc 1m 51s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 23s the patch passed with JDK v1.7.0_91 +1 unit 0m 19s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 56s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 79m 35s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 10s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 80m 43s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 226m 56s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780433/YARN-4265-trunk.002.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 14719e8d400e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 778146e Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10149/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10149/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu for reply and updating the patch.
          I am still reviewing the new patch but have some quick response below:

          Plugins are provided by third-party applications such as Tez. Right now we cannot assume which exact entity group plugin the user is using, therefore we have to conservatively leave this config as empty.

          Thanks for clarification on this. Can we provide a default one (could be simple) for built-in YARN applications, i.e. distributed shell? This can be helpful to demonstrate how to use it for applications other than Tez. Or other apps have to refer Tez code on how to use it. IMO, this is not a very good practice for YARN.

          we're using fine-grained locking for each cached item in the reader cache, and cache refresh only happens infrequently (~10 secs by default), so maybe we'd like to stabilize the whole synchronization story before fine tune this part?

          Sounds like a plan. Looks like we still need to make effort on checking the whole synchronization mechanism. Isn't it?

          Show
          djp Junping Du added a comment - Thanks Li Lu for reply and updating the patch. I am still reviewing the new patch but have some quick response below: Plugins are provided by third-party applications such as Tez. Right now we cannot assume which exact entity group plugin the user is using, therefore we have to conservatively leave this config as empty. Thanks for clarification on this. Can we provide a default one (could be simple) for built-in YARN applications, i.e. distributed shell? This can be helpful to demonstrate how to use it for applications other than Tez. Or other apps have to refer Tez code on how to use it. IMO, this is not a very good practice for YARN. we're using fine-grained locking for each cached item in the reader cache, and cache refresh only happens infrequently (~10 secs by default), so maybe we'd like to stabilize the whole synchronization story before fine tune this part? Sounds like a plan. Looks like we still need to make effort on checking the whole synchronization mechanism. Isn't it?
          Hide
          gtCarrera9 Li Lu added a comment -

          Can we provide a default one (could be simple) for built-in YARN applications, i.e. distributed shell?

          We can certainly do that, given that the new APIs are in and we've got this patch. Since this is a separate work than introducing the whole new storage (as in this JIRA), maybe we can address this as a new JIRA? I can start off doing that right away.

          Looks like we still need to make effort on checking the whole synchronization mechanism. Isn't it?

          I checked all synchronization logic a few times before I submit the original patch. We're using fine-grained locking in this patch, so yes it's more error pruning. I'll check again later on and fix any problem I discovered. Feel free to raise up any concerns in your review. Thanks!

          Show
          gtCarrera9 Li Lu added a comment - Can we provide a default one (could be simple) for built-in YARN applications, i.e. distributed shell? We can certainly do that, given that the new APIs are in and we've got this patch. Since this is a separate work than introducing the whole new storage (as in this JIRA), maybe we can address this as a new JIRA? I can start off doing that right away. Looks like we still need to make effort on checking the whole synchronization mechanism. Isn't it? I checked all synchronization logic a few times before I submit the original patch. We're using fine-grained locking in this patch, so yes it's more error pruning. I'll check again later on and fix any problem I discovered. Feel free to raise up any concerns in your review. Thanks!
          Hide
          gtCarrera9 Li Lu added a comment -

          Refresh the patch once more to:

          1. Add documentations about cache item and its main features
          2. Double check synchronizations and fix unsynchronized accessors.
          3. Change accessor modifiers to some methods to make them stricter.

          Ran findbugs locally and it looks fine.

          Show
          gtCarrera9 Li Lu added a comment - Refresh the patch once more to: Add documentations about cache item and its main features Double check synchronizations and fix unsynchronized accessors. Change accessor modifiers to some methods to make them stricter. Ran findbugs locally and it looks fine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 8m 19s trunk passed
          +1 compile 2m 2s trunk passed with JDK v1.8.0_66
          +1 compile 2m 26s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 2m 40s trunk passed
          +1 mvneclipse 0m 48s trunk passed
          -1 findbugs 3m 25s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          +1 javadoc 2m 14s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 50s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 29s the patch passed
          +1 compile 2m 6s the patch passed with JDK v1.8.0_66
          +1 javac 2m 6s the patch passed
          +1 compile 2m 25s the patch passed with JDK v1.7.0_91
          +1 javac 2m 25s the patch passed
          -1 checkstyle 0m 33s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300).
          +1 mvnsite 2m 47s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 3m 41s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 3m 58s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 28 new issues (was 544, now 552).
          +1 javadoc 2m 10s the patch passed with JDK v1.8.0_66
          -1 javadoc 9m 4s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 3, now 4).
          +1 javadoc 4m 43s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 77m 24s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 10s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 79m 38s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          218m 47s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780628/YARN-4265-trunk.003.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 1abf81a2424a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dec8fed
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10161/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10161/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 8m 19s trunk passed +1 compile 2m 2s trunk passed with JDK v1.8.0_66 +1 compile 2m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 35s trunk passed +1 mvnsite 2m 40s trunk passed +1 mvneclipse 0m 48s trunk passed -1 findbugs 3m 25s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) +1 javadoc 2m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 50s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 29s the patch passed +1 compile 2m 6s the patch passed with JDK v1.8.0_66 +1 javac 2m 6s the patch passed +1 compile 2m 25s the patch passed with JDK v1.7.0_91 +1 javac 2m 25s the patch passed -1 checkstyle 0m 33s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300). +1 mvnsite 2m 47s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 3m 41s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 3m 58s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 28 new issues (was 544, now 552). +1 javadoc 2m 10s the patch passed with JDK v1.8.0_66 -1 javadoc 9m 4s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 3, now 4). +1 javadoc 4m 43s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 77m 24s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 10s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 79m 38s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 218m 47s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780628/YARN-4265-trunk.003.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1abf81a2424a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dec8fed Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10161/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10161/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10161/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu for updating the patch.

          Since this is a separate work than introducing the whole new storage (as in this JIRA), maybe we can address this as a new JIRA?

          +1. Address this in a separated JIRA sounds good to me, and I can take a look at that demo patch in reviewing this JIRA.

          Show
          djp Junping Du added a comment - Thanks Li Lu for updating the patch. Since this is a separate work than introducing the whole new storage (as in this JIRA), maybe we can address this as a new JIRA? +1. Address this in a separated JIRA sounds good to me, and I can take a look at that demo patch in reviewing this JIRA.
          Hide
          gtCarrera9 Li Lu added a comment -

          Sure. Linking this the YARN-4545 which works on the distributed shell integration.

          Show
          gtCarrera9 Li Lu added a comment - Sure. Linking this the YARN-4545 which works on the distributed shell integration.
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu for updating the patch.
          trunk.003 patch is getting closer, more comments:
          In yarn-default.xml,

          +      How long the ATS v1.5 entity group file system storage systemwill keep an
          +      application's data in the done directory.
          

          "file system storage systemwill" should be "file system storage will"

          In EntityCacheItem.java, inside of refreshCache():

          +    if (needRefresh()) {
          +      if (!appLogs.isDone()) {
          +        appLogs.parseSummaryLogs();
          +      } else if (appLogs.getDetailLogs().isEmpty()) {
          +        appLogs.scanForLogs();
          +      }
          +      if (!appLogs.getDetailLogs().isEmpty()) {
          

          I am a bit confused with logic here: if appLogs is not done yet, but its detail logs is empty, do we need to scanForLogs? If not, we should document the reason at the least.

          +        if (appLogs.getDetailLogs().isEmpty()) {
          +          LOG.debug("cache id {}'s detail log is empty! ", groupId);
          +        }
          

          This is inside of case {{ if (!appLogs.getDetailLogs().isEmpty()) }}, does it possible for detailLogs to become empty again? If not, we should remove it?

          +          // Only refresh the log that matches the cache id
          +          if (log.getFilename().contains(groupId.toString())) {
          

          If we have two groupIds: 114859476_01_1 and 114859476_01_11, the later one's log file name can match with previous groupId as well? If so, we may consider to match file name with cache id more exactly? The same case with code below {{if (log.getFilename().contains(groupId.toString())) }}

          appLogs.getDetailLogs().removeAll(removeList); should be move out of for each, or removeList will be useless here.

          In EntityGroupFSTimelineStore.java,

          +          + "%04d" + Path.SEPARATOR // app num / 10000000
          

          I think it should be app num / 1000,000 (1 million) but not 10 millions.

          +  private final Map<TimelineEntityGroupId, EntityCacheItem> cachedLogs
          +      = Collections.synchronizedMap(
          +      new LinkedHashMap<TimelineEntityGroupId, EntityCacheItem>(
          +          appCacheMaxSize + 1, 0.75f, true) {
          

          It should move cachedLogs initiaization into serviceInit after appCacheMaxSize is setup already.

          +  @SuppressWarnings("serial")
          +  @Override
          +  protected void serviceInit(Configuration conf) throws Exception {
          

          SuppressWarnings("serial") seems to be unnecessary?

          In loadPlugIns(),

          +      } catch (Exception e) {
          +        LOG.info("Error loading plugin " + name, e);
          +      }
          

          It should use warn instead of info level for log message.

          +      LOG.debug("Load plugin class {}", cacheIdPlugin.getClass().getName());
          

          Better to use info instead of debug to follow the same practice with other messages in serviceInit.

          For cleanLogs(Path dirpath), it seems like the execution result of cleanup log depends on the order of files/directories returned. Say an app dir include: file A, dir B, file A is a fresh one and all files in dir B are older than logRetainMillis. If file A get return first, the cleanLogs() do nothing, but if dir B get return first, cleanLogs() will clenup dir B. Give fs.listStatusIterator(dirpath) could return file A, dir B in randomly order, is this randomly behavior expected?

              // scans for new logs and returns the modification timestamp of the
              // most recently modified log
              @InterfaceAudience.Private
              @VisibleForTesting
              long scanForLogs() throws IOException {
          

          Can we directly return appDirPath's modification time instead of go through all sub directories? Besides domain, summary, entity logs, anything else could exist under this directory?

                for (LogInfo log : summaryLogs) {
                  if (log.getFilename().equals(filename)
                      && log.getAttemptDirName().equals(attemptDirName)) {
                    return;
                  }
                }
          

          Is it a common case for a AppLogs have many summaryLogs (and detail logs)? I guess not, but if my guess is wrong, we may should to use hashmap instead of list which costs to much time in search.

          In LogInfo.parsePath(),

              try {
                in.seek(offset);
                try {
                  parser = jsonFactory.createJsonParser(in);
                  parser.configure(JsonParser.Feature.AUTO_CLOSE_SOURCE, false);
                } catch (IOException e) {
                  // if app hasn't completed then there may be errors due to the
                  // incomplete file which are treated as EOF until app completes
                  if (appCompleted) {
                    throw e;
                  } else {
                    LOG.debug("Exception in parse path: {}", e.getMessage());
                  }
                }
                return doParse(tdm, parser, objMapper, ugi, appCompleted);
          

          Even we get rid of throw exception for app incompleted case, doParse() below will throw a NPE exception. Is this expected?

          In PluginStoreTestUtils.java,

            static TimelineEntities generateTestEntities() {
                  TimelineEntities entities = new TimelineEntities();
          

          Indentation issue here.

          Show
          djp Junping Du added a comment - Thanks Li Lu for updating the patch. trunk.003 patch is getting closer, more comments: In yarn-default.xml, + How long the ATS v1.5 entity group file system storage systemwill keep an + application's data in the done directory. "file system storage systemwill" should be "file system storage will" In EntityCacheItem.java, inside of refreshCache(): + if (needRefresh()) { + if (!appLogs.isDone()) { + appLogs.parseSummaryLogs(); + } else if (appLogs.getDetailLogs().isEmpty()) { + appLogs.scanForLogs(); + } + if (!appLogs.getDetailLogs().isEmpty()) { I am a bit confused with logic here: if appLogs is not done yet, but its detail logs is empty, do we need to scanForLogs? If not, we should document the reason at the least. + if (appLogs.getDetailLogs().isEmpty()) { + LOG.debug( "cache id {}'s detail log is empty! " , groupId); + } This is inside of case {{ if (!appLogs.getDetailLogs().isEmpty()) }}, does it possible for detailLogs to become empty again? If not, we should remove it? + // Only refresh the log that matches the cache id + if (log.getFilename().contains(groupId.toString())) { If we have two groupIds: 114859476_01_1 and 114859476_01_11, the later one's log file name can match with previous groupId as well? If so, we may consider to match file name with cache id more exactly? The same case with code below {{if (log.getFilename().contains(groupId.toString())) }} appLogs.getDetailLogs().removeAll(removeList); should be move out of for each, or removeList will be useless here. In EntityGroupFSTimelineStore.java, + + "%04d" + Path.SEPARATOR // app num / 10000000 I think it should be app num / 1000,000 (1 million) but not 10 millions. + private final Map<TimelineEntityGroupId, EntityCacheItem> cachedLogs + = Collections.synchronizedMap( + new LinkedHashMap<TimelineEntityGroupId, EntityCacheItem>( + appCacheMaxSize + 1, 0.75f, true ) { It should move cachedLogs initiaization into serviceInit after appCacheMaxSize is setup already. + @SuppressWarnings( "serial" ) + @Override + protected void serviceInit(Configuration conf) throws Exception { SuppressWarnings("serial") seems to be unnecessary? In loadPlugIns(), + } catch (Exception e) { + LOG.info( "Error loading plugin " + name, e); + } It should use warn instead of info level for log message. + LOG.debug( "Load plugin class {}" , cacheIdPlugin.getClass().getName()); Better to use info instead of debug to follow the same practice with other messages in serviceInit. For cleanLogs(Path dirpath), it seems like the execution result of cleanup log depends on the order of files/directories returned. Say an app dir include: file A, dir B, file A is a fresh one and all files in dir B are older than logRetainMillis. If file A get return first, the cleanLogs() do nothing, but if dir B get return first, cleanLogs() will clenup dir B. Give fs.listStatusIterator(dirpath) could return file A, dir B in randomly order, is this randomly behavior expected? // scans for new logs and returns the modification timestamp of the // most recently modified log @InterfaceAudience.Private @VisibleForTesting long scanForLogs() throws IOException { Can we directly return appDirPath's modification time instead of go through all sub directories? Besides domain, summary, entity logs, anything else could exist under this directory? for (LogInfo log : summaryLogs) { if (log.getFilename().equals(filename) && log.getAttemptDirName().equals(attemptDirName)) { return ; } } Is it a common case for a AppLogs have many summaryLogs (and detail logs)? I guess not, but if my guess is wrong, we may should to use hashmap instead of list which costs to much time in search. In LogInfo.parsePath(), try { in.seek(offset); try { parser = jsonFactory.createJsonParser(in); parser.configure(JsonParser.Feature.AUTO_CLOSE_SOURCE, false ); } catch (IOException e) { // if app hasn't completed then there may be errors due to the // incomplete file which are treated as EOF until app completes if (appCompleted) { throw e; } else { LOG.debug( "Exception in parse path: {}" , e.getMessage()); } } return doParse(tdm, parser, objMapper, ugi, appCompleted); Even we get rid of throw exception for app incompleted case, doParse() below will throw a NPE exception. Is this expected? In PluginStoreTestUtils.java, static TimelineEntities generateTestEntities() { TimelineEntities entities = new TimelineEntities(); Indentation issue here.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Junping Du for the review! I updated my patch according to your comments. Some quick comments:

          I am a bit confused with logic here: if appLogs is not done yet, but its detail logs is empty, do we need to scanForLogs? If not, we should document the reason at the least.

          Yes, we only update summary logs when the app is running. Updated comments for this.

          If we have two groupIds: 114859476_01_1 and 114859476_01_11, the later one's log file name can match with previous groupId as well? If so, we may consider to match file name with cache id more exactly? The same case with code below {{if (log.getFilename().contains(groupId.toString())) }}

          Nice catch! What I'm trying to address here is the names with entity group id and a sequence number. I've updated related logic here.

          For cleanLogs(Path dirpath), it seems like the execution result of cleanup log depends on the order of files/directories returned. Say an app dir include: file A, dir B, file A is a fresh one and all files in dir B are older than logRetainMillis. If file A get return first, the cleanLogs() do nothing, but if dir B get return first, cleanLogs() will clenup dir B. Give fs.listStatusIterator(dirpath) could return file A, dir B in randomly order, is this randomly behavior expected?

          This is not possible because in the first part of cleanLogs(), we're only doing a DFS to decide if we need to remove this dir. If any file in the directory is new, we will not remove it. The detailed remove logic happens after the DFS process.

          Is it a common case for a AppLogs have many summaryLogs (and detail logs)?

          Right now we're not facing this kind of use case. We can certainly optimize this logic in future though.

          Can we directly return appDirPath's modification time instead of go through all sub directories?

          I believe we cannot. We're trying to return the latest time any file within a directory has been changed to decide if the app is in UNKNOWN state for long enough in parseSummaryLogs.

          Show
          gtCarrera9 Li Lu added a comment - Thanks Junping Du for the review! I updated my patch according to your comments. Some quick comments: I am a bit confused with logic here: if appLogs is not done yet, but its detail logs is empty, do we need to scanForLogs? If not, we should document the reason at the least. Yes, we only update summary logs when the app is running. Updated comments for this. If we have two groupIds: 114859476_01_1 and 114859476_01_11, the later one's log file name can match with previous groupId as well? If so, we may consider to match file name with cache id more exactly? The same case with code below {{if (log.getFilename().contains(groupId.toString())) }} Nice catch! What I'm trying to address here is the names with entity group id and a sequence number. I've updated related logic here. For cleanLogs(Path dirpath), it seems like the execution result of cleanup log depends on the order of files/directories returned. Say an app dir include: file A, dir B, file A is a fresh one and all files in dir B are older than logRetainMillis. If file A get return first, the cleanLogs() do nothing, but if dir B get return first, cleanLogs() will clenup dir B. Give fs.listStatusIterator(dirpath) could return file A, dir B in randomly order, is this randomly behavior expected? This is not possible because in the first part of cleanLogs(), we're only doing a DFS to decide if we need to remove this dir. If any file in the directory is new, we will not remove it. The detailed remove logic happens after the DFS process. Is it a common case for a AppLogs have many summaryLogs (and detail logs)? Right now we're not facing this kind of use case. We can certainly optimize this logic in future though. Can we directly return appDirPath's modification time instead of go through all sub directories? I believe we cannot. We're trying to return the latest time any file within a directory has been changed to decide if the app is in UNKNOWN state for long enough in parseSummaryLogs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 7m 27s trunk passed
          +1 compile 1m 45s trunk passed with JDK v1.8.0_66
          +1 compile 2m 3s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 2m 27s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          -1 findbugs 3m 16s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 0m 44s hadoop-yarn-server in trunk failed with JDK v1.8.0_66.
          +1 javadoc 4m 21s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 4s the patch passed
          +1 compile 1m 46s the patch passed with JDK v1.8.0_66
          +1 javac 1m 46s the patch passed
          +1 compile 2m 6s the patch passed with JDK v1.7.0_91
          +1 javac 2m 6s the patch passed
          -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300).
          +1 mvnsite 2m 29s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 3m 23s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 0m 42s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          -1 javadoc 7m 54s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 3, now 4).
          +1 javadoc 4m 18s the patch passed with JDK v1.7.0_91
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 72m 13s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 75m 2s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          202m 18s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781349/YARN-4265-trunk.004.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e01d1240fbac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 109e528
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10217/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10217/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 7m 27s trunk passed +1 compile 1m 45s trunk passed with JDK v1.8.0_66 +1 compile 2m 3s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 27s trunk passed +1 mvneclipse 0m 43s trunk passed -1 findbugs 3m 16s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 0m 44s hadoop-yarn-server in trunk failed with JDK v1.8.0_66. +1 javadoc 4m 21s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 4s the patch passed +1 compile 1m 46s the patch passed with JDK v1.8.0_66 +1 javac 1m 46s the patch passed +1 compile 2m 6s the patch passed with JDK v1.7.0_91 +1 javac 2m 6s the patch passed -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300). +1 mvnsite 2m 29s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 3m 23s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 0m 42s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. -1 javadoc 7m 54s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 3, now 4). +1 javadoc 4m 18s the patch passed with JDK v1.7.0_91 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 50s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 72m 13s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 75m 2s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 202m 18s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781349/YARN-4265-trunk.004.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e01d1240fbac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 109e528 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10217/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10217/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10217/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu for reply my comments above.

          This is not possible because in the first part of cleanLogs(), we're only doing a DFS to decide if we need to remove this dir. If any file in the directory is new, we will not remove it. The detailed remove logic happens after the DFS process.

          If our intention is to a DFS (deep first search), then I would say the implementation of cleanLogs() failed to achieve this because the flag of shouldClean in recursively calling cleanLogs() doesn't passed up to upper caller to reflect shouldClean flag there. We should fix it with a unit test.

          I believe we cannot. We're trying to return the latest time any file within a directory has been changed to decide if the app is in UNKNOWN state for long enough in parseSummaryLogs.

          For HDFS, any file update should reflect the modification time to parent directory. Any exception case here?

          Show
          djp Junping Du added a comment - Thanks Li Lu for reply my comments above. This is not possible because in the first part of cleanLogs(), we're only doing a DFS to decide if we need to remove this dir. If any file in the directory is new, we will not remove it. The detailed remove logic happens after the DFS process. If our intention is to a DFS (deep first search), then I would say the implementation of cleanLogs() failed to achieve this because the flag of shouldClean in recursively calling cleanLogs() doesn't passed up to upper caller to reflect shouldClean flag there. We should fix it with a unit test. I believe we cannot. We're trying to return the latest time any file within a directory has been changed to decide if the app is in UNKNOWN state for long enough in parseSummaryLogs. For HDFS, any file update should reflect the modification time to parent directory. Any exception case here?
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Junping Du and Li Lu,

          For HDFS, any file update should reflect the modification time to parent directory. Any exception case here?

          For creating/moving files in an existing directory, the modification time of the parent directory will be updated. Refer to code here.

          However, should this jira consider appending to an existing file, the modification time of the parent directory will not be updated. Refer to code here.

          Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Hi Junping Du and Li Lu , For HDFS, any file update should reflect the modification time to parent directory. Any exception case here? For creating/moving files in an existing directory, the modification time of the parent directory will be updated. Refer to code here . However, should this jira consider appending to an existing file, the modification time of the parent directory will not be updated. Refer to code here . Thanks.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Junping Du and Mingliang Liu for the help! Since we need to handle files with appends, we cannot directly use directory modification time to decide if the contents of a directory has been changed. This means we also need to change some logics in the cleanLogs method. I redesigned cleanLogs method to perform a log scan with two methods:
          Method 1: For the given directory, search (in a depth first fashion) to find out application log directories. For each of them, call method 2.
          Method 2: For the given application log directory, search all files inside. If there exists a file that has been "recently" (as defined by the configs) updates, skip removing this directory. Otherwise, remove this application log directory.

          In this way we can search inside a directory for all application log directories that need to be reclaimed.

          According to Junping's suggestion, I've also added a new unit test (testCleanLogs) to cover common cases for the cleanLogs method.

          Show
          gtCarrera9 Li Lu added a comment - Thanks Junping Du and Mingliang Liu for the help! Since we need to handle files with appends, we cannot directly use directory modification time to decide if the contents of a directory has been changed. This means we also need to change some logics in the cleanLogs method. I redesigned cleanLogs method to perform a log scan with two methods: Method 1: For the given directory, search (in a depth first fashion) to find out application log directories. For each of them, call method 2. Method 2: For the given application log directory, search all files inside. If there exists a file that has been "recently" (as defined by the configs) updates, skip removing this directory. Otherwise, remove this application log directory. In this way we can search inside a directory for all application log directories that need to be reclaimed. According to Junping's suggestion, I've also added a new unit test (testCleanLogs) to cover common cases for the cleanLogs method.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 7m 34s trunk passed
          +1 compile 1m 52s trunk passed with JDK v1.8.0_66
          +1 compile 2m 6s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 2m 30s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          -1 findbugs 3m 17s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 0m 44s hadoop-yarn-server in trunk failed with JDK v1.8.0_66.
          +1 javadoc 4m 29s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 7s the patch passed
          +1 compile 1m 50s the patch passed with JDK v1.8.0_66
          +1 javac 1m 50s the patch passed
          +1 compile 2m 8s the patch passed with JDK v1.7.0_91
          +1 javac 2m 8s the patch passed
          -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300).
          +1 mvnsite 2m 28s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 3m 25s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 1m 3s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          -1 javadoc 8m 49s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 3, now 4).
          +1 javadoc 4m 32s the patch passed with JDK v1.7.0_91
          +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 14s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 72m 10s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 7s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 74m 7s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          203m 27s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781700/YARN-4265-trunk.005.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e2474be5d70e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c2e2e13
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10237/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10237/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 7m 34s trunk passed +1 compile 1m 52s trunk passed with JDK v1.8.0_66 +1 compile 2m 6s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 30s trunk passed +1 mvneclipse 0m 42s trunk passed -1 findbugs 3m 17s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 0m 44s hadoop-yarn-server in trunk failed with JDK v1.8.0_66. +1 javadoc 4m 29s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 7s the patch passed +1 compile 1m 50s the patch passed with JDK v1.8.0_66 +1 javac 1m 50s the patch passed +1 compile 2m 8s the patch passed with JDK v1.7.0_91 +1 javac 2m 8s the patch passed -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300). +1 mvnsite 2m 28s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 3m 25s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 1m 3s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. -1 javadoc 8m 49s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 3, now 4). +1 javadoc 4m 32s the patch passed with JDK v1.7.0_91 +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 14s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 72m 10s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 7s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 74m 7s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 203m 27s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781700/YARN-4265-trunk.005.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e2474be5d70e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c2e2e13 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10237/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10237/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu for patient and updating the patch.
          A few minor comments:
          For new algorithm in cleanLogs(), I think it only works for specific log directory/file structure. Isn't it? If so, can you document it as javadoc on specific log structure it applies on?
          For new added test case, please correct some naming issues with DIRs in each case. Also, instead of calling fs.getFileStatus(appDirClean) and catch exceptions for file/directory non-existing, it is more convenient to use fs.exists(Path).
          Please correct javadoc warning reported by Jenkins also.

          Show
          djp Junping Du added a comment - Thanks Li Lu for patient and updating the patch. A few minor comments: For new algorithm in cleanLogs(), I think it only works for specific log directory/file structure. Isn't it? If so, can you document it as javadoc on specific log structure it applies on? For new added test case, please correct some naming issues with DIRs in each case. Also, instead of calling fs.getFileStatus(appDirClean) and catch exceptions for file/directory non-existing, it is more convenient to use fs.exists(Path) . Please correct javadoc warning reported by Jenkins also.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Junping Du for the review! I addressed all of you concerns in the latest patch. Any other commnets? cc/Xuan Gong, Jason Lowe

          Show
          gtCarrera9 Li Lu added a comment - Thanks Junping Du for the review! I addressed all of you concerns in the latest patch. Any other commnets? cc/ Xuan Gong , Jason Lowe
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 34s Maven dependency ordering for branch
          +1 mvninstall 7m 27s trunk passed
          +1 compile 1m 53s trunk passed with JDK v1.8.0_66
          +1 compile 2m 7s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 2m 52s trunk passed
          +1 mvneclipse 0m 55s trunk passed
          -1 findbugs 3m 16s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          +1 javadoc 2m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 48s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 43s Maven dependency ordering for patch
          +1 mvninstall 2m 47s the patch passed
          +1 compile 1m 54s the patch passed with JDK v1.8.0_66
          +1 javac 1m 54s the patch passed
          +1 compile 2m 11s the patch passed with JDK v1.7.0_91
          +1 javac 2m 11s the patch passed
          -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300).
          +1 mvnsite 3m 19s the patch passed
          +1 mvneclipse 1m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 3m 58s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 4m 1s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 544, now 551).
          -1 javadoc 4m 1s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200).
          -1 javadoc 4m 1s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200).
          +1 javadoc 2m 53s the patch passed with JDK v1.8.0_66
          +1 javadoc 5m 20s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 11s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 16m 33s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 3m 10s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 16s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 27s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 76m 3s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 unit 3m 26s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 unit 3m 21s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          172m 55s



          Reason Tests
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782144/YARN-4265-trunk.006.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2ee81e11226f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 321072b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10269/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10269/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 34s Maven dependency ordering for branch +1 mvninstall 7m 27s trunk passed +1 compile 1m 53s trunk passed with JDK v1.8.0_66 +1 compile 2m 7s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 52s trunk passed +1 mvneclipse 0m 55s trunk passed -1 findbugs 3m 16s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) +1 javadoc 2m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 48s trunk passed with JDK v1.7.0_91 0 mvndep 0m 43s Maven dependency ordering for patch +1 mvninstall 2m 47s the patch passed +1 compile 1m 54s the patch passed with JDK v1.8.0_66 +1 javac 1m 54s the patch passed +1 compile 2m 11s the patch passed with JDK v1.7.0_91 +1 javac 2m 11s the patch passed -1 checkstyle 0m 31s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 300). +1 mvnsite 3m 19s the patch passed +1 mvneclipse 1m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 3m 58s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 4m 1s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 544, now 551). -1 javadoc 4m 1s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200). -1 javadoc 4m 1s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200). +1 javadoc 2m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 5m 20s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 11s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 16m 33s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 3m 10s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 3m 16s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 27s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 76m 3s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 unit 3m 26s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 unit 3m 21s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 172m 55s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782144/YARN-4265-trunk.006.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2ee81e11226f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 321072b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10269/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10269/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10269/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          New patch adds the missing package-info.java that the check-style script is mourning about.

          Show
          gtCarrera9 Li Lu added a comment - New patch adds the missing package-info.java that the check-style script is mourning about.
          Hide
          gtCarrera9 Li Lu added a comment -

          Also, the failing unit tests are not related to the changes in this JIRA.

          Show
          gtCarrera9 Li Lu added a comment - Also, the failing unit tests are not related to the changes in this JIRA.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 50s Maven dependency ordering for branch
          +1 mvninstall 7m 41s trunk passed
          +1 compile 1m 46s trunk passed with JDK v1.8.0_66
          +1 compile 2m 7s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 2m 55s trunk passed
          +1 mvneclipse 0m 58s trunk passed
          -1 findbugs 3m 16s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 59s trunk passed with JDK v1.7.0_91
          -1 mvndep 0m 27s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server dependency:list failed
          -1 mvndep 0m 44s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage dependency:list failed
          0 mvndep 0m 44s Maven dependency ordering for patch
          +1 mvninstall 2m 48s the patch passed
          +1 compile 1m 48s the patch passed with JDK v1.8.0_66
          +1 javac 1m 48s the patch passed
          +1 compile 2m 8s the patch passed with JDK v1.7.0_91
          +1 javac 2m 8s the patch passed
          -1 checkstyle 0m 33s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 293, now 301).
          +1 mvnsite 3m 21s the patch passed
          +1 mvneclipse 1m 5s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 3m 32s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml)
          -1 javadoc 3m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 544, now 551).
          -1 javadoc 3m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200).
          -1 javadoc 3m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200).
          +1 javadoc 2m 21s the patch passed with JDK v1.8.0_66
          +1 javadoc 5m 5s the patch passed with JDK v1.7.0_91
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 78m 36s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 3m 41s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 43s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 79m 59s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 unit 4m 8s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 unit 4m 5s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          239m 23s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782186/YARN-4265-trunk.007.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 44a210f396e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8315582
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10273/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10273/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 50s Maven dependency ordering for branch +1 mvninstall 7m 41s trunk passed +1 compile 1m 46s trunk passed with JDK v1.8.0_66 +1 compile 2m 7s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 31s trunk passed +1 mvnsite 2m 55s trunk passed +1 mvneclipse 0m 58s trunk passed -1 findbugs 3m 16s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 59s trunk passed with JDK v1.7.0_91 -1 mvndep 0m 27s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server dependency:list failed -1 mvndep 0m 44s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage dependency:list failed 0 mvndep 0m 44s Maven dependency ordering for patch +1 mvninstall 2m 48s the patch passed +1 compile 1m 48s the patch passed with JDK v1.8.0_66 +1 javac 1m 48s the patch passed +1 compile 2m 8s the patch passed with JDK v1.7.0_91 +1 javac 2m 8s the patch passed -1 checkstyle 0m 33s Patch generated 9 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 293, now 301). +1 mvnsite 3m 21s the patch passed +1 mvneclipse 1m 5s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 3m 32s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/target/findbugsXml.xml) -1 javadoc 3m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 544, now 551). -1 javadoc 3m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200). -1 javadoc 3m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200). +1 javadoc 2m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 5m 5s the patch passed with JDK v1.7.0_91 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 53s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 78m 36s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 3m 41s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 3m 43s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 79m 59s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 unit 4m 8s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 unit 4m 5s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 239m 23s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782186/YARN-4265-trunk.007.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 44a210f396e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8315582 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10273/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10273/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10273/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          The checkstyle complain:

          ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/LogInfo.java:90:    String filename = getFilename();:12: 'filename' hides a field"
          

          should be taken care of. Also, it sounds like maven dependency failed?

          Show
          djp Junping Du added a comment - The checkstyle complain: ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/LogInfo.java:90: String filename = getFilename();:12: 'filename' hides a field" should be taken care of. Also, it sounds like maven dependency failed?
          Hide
          gtCarrera9 Li Lu added a comment -

          New patch to address the checkstyle hidden field warning. I also tested locally for the maven dependency issue. I cleaned my local m2 repository, rebuilt hadoop, and ran maven dependency. I cannot reproduce the problem. Also, the problem appears to be intermittent in the past Jenkins runs.

          Show
          gtCarrera9 Li Lu added a comment - New patch to address the checkstyle hidden field warning. I also tested locally for the maven dependency issue. I cleaned my local m2 repository, rebuilt hadoop, and ran maven dependency. I cannot reproduce the problem. Also, the problem appears to be intermittent in the past Jenkins runs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 38s Maven dependency ordering for branch
          +1 mvninstall 8m 39s trunk passed
          +1 compile 2m 31s trunk passed with JDK v1.8.0_66
          +1 compile 2m 25s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 3m 11s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 2m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 5m 2s trunk passed with JDK v1.7.0_91
          -1 mvndep 0m 27s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server dependency:list failed
          -1 mvndep 0m 44s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage dependency:list failed
          0 mvndep 0m 44s Maven dependency ordering for patch
          +1 mvninstall 2m 58s the patch passed
          +1 compile 2m 3s the patch passed with JDK v1.8.0_66
          +1 javac 2m 3s the patch passed
          +1 compile 2m 23s the patch passed with JDK v1.7.0_91
          +1 javac 2m 23s the patch passed
          -1 checkstyle 0m 34s Patch generated 8 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 299).
          +1 mvnsite 3m 32s the patch passed
          +1 mvneclipse 1m 7s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          +1 findbugs 4m 34s the patch passed
          -1 javadoc 3m 39s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 544, now 551).
          -1 javadoc 3m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200).
          -1 javadoc 3m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200).
          +1 javadoc 2m 35s the patch passed with JDK v1.8.0_66
          +1 javadoc 5m 17s the patch passed with JDK v1.7.0_91
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 0s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 74m 3s hadoop-yarn-server in the patch failed with JDK v1.8.0_66.
          +1 unit 3m 8s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 3m 9s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 18s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          -1 unit 75m 59s hadoop-yarn-server in the patch failed with JDK v1.7.0_91.
          +1 unit 3m 20s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 unit 3m 19s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          225m 51s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782321/YARN-4265-trunk.008.patch
          JIRA Issue YARN-4265
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 360a943f7467 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9d04f26
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10286/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10286/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 38s Maven dependency ordering for branch +1 mvninstall 8m 39s trunk passed +1 compile 2m 31s trunk passed with JDK v1.8.0_66 +1 compile 2m 25s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 33s trunk passed +1 mvnsite 3m 11s trunk passed +1 mvneclipse 0m 59s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server +1 findbugs 3m 26s trunk passed +1 javadoc 2m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 5m 2s trunk passed with JDK v1.7.0_91 -1 mvndep 0m 27s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server dependency:list failed -1 mvndep 0m 44s patch's hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage dependency:list failed 0 mvndep 0m 44s Maven dependency ordering for patch +1 mvninstall 2m 58s the patch passed +1 compile 2m 3s the patch passed with JDK v1.8.0_66 +1 javac 2m 3s the patch passed +1 compile 2m 23s the patch passed with JDK v1.7.0_91 +1 javac 2m 23s the patch passed -1 checkstyle 0m 34s Patch generated 8 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 292, now 299). +1 mvnsite 3m 32s the patch passed +1 mvneclipse 1m 7s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server +1 findbugs 4m 34s the patch passed -1 javadoc 3m 39s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 544, now 551). -1 javadoc 3m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200). -1 javadoc 3m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66 with JDK v1.8.0_66 generated 100 new issues (was 100, now 200). +1 javadoc 2m 35s the patch passed with JDK v1.8.0_66 +1 javadoc 5m 17s the patch passed with JDK v1.7.0_91 +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 0s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 74m 3s hadoop-yarn-server in the patch failed with JDK v1.8.0_66. +1 unit 3m 8s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 3m 9s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 18s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. -1 unit 75m 59s hadoop-yarn-server in the patch failed with JDK v1.7.0_91. +1 unit 3m 20s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 unit 3m 19s hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 225m 51s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782321/YARN-4265-trunk.008.patch JIRA Issue YARN-4265 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 360a943f7467 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9d04f26 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt mvndep hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/maven-patch-dependencylist-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10286/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10286/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10286/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          The maven dependency check uses a separate maven repository -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-0. I tested the whole workflow for building from the scratch and it worked. The AHS-test jar has not been published to apache SNAPSHOT (timeline plugin is the first module to depend on this module) yet. Therefore the mvn dependency test appears to be unrelated here.

          Show
          gtCarrera9 Li Lu added a comment - The maven dependency check uses a separate maven repository -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-0 . I tested the whole workflow for building from the scratch and it worked. The AHS-test jar has not been published to apache SNAPSHOT (timeline plugin is the first module to depend on this module) yet. Therefore the mvn dependency test appears to be unrelated here.
          Hide
          djp Junping Du added a comment -

          Tried mvn dependency:list locally for hadoop-yarn-server-timeline-pluginstorage and hadoop-yarn-server, both get passed successfully. Agree that test failures are also unrelated. Looks like all my major comments has been addressed.
          Jason Lowe and Xuan Gong, do you have further comments here?

          Show
          djp Junping Du added a comment - Tried mvn dependency:list locally for hadoop-yarn-server-timeline-pluginstorage and hadoop-yarn-server , both get passed successfully. Agree that test failures are also unrelated. Looks like all my major comments has been addressed. Jason Lowe and Xuan Gong , do you have further comments here?
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm.

          Show
          jlowe Jason Lowe added a comment - +1 lgtm.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9128/)
          YARN-4265. Provide new timeline plugin storage to support fine-grained (junping_du: rev 02f597c5db36ded385413958bdee793ad7eda40e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/package-info.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineEntityGroupPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityCacheItem.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLogInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/PluginStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/EntityGroupPlugInForTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/LogInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManagerMetrics.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9128/ ) YARN-4265 . Provide new timeline plugin storage to support fine-grained (junping_du: rev 02f597c5db36ded385413958bdee793ad7eda40e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineEntityGroupPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityCacheItem.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLogInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/PluginStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/EntityGroupPlugInForTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/LogInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManagerMetrics.java
          Hide
          djp Junping Du added a comment -

          Thanks Li Lu and Jason Lowe for the patch contribution! I have commit the 08 patch to trunk, branch-2 and branch-2.8.

          Show
          djp Junping Du added a comment - Thanks Li Lu and Jason Lowe for the patch contribution! I have commit the 08 patch to trunk, branch-2 and branch-2.8.
          Hide
          kasha Karthik Kambatla added a comment -

          This seems to break branch-2.8 compilation (mvn clean install -DskipTests). Can someone look into it?

          Show
          kasha Karthik Kambatla added a comment - This seems to break branch-2.8 compilation (mvn clean install -DskipTests). Can someone look into it?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I found the issue, failure is because newly added project pom.xml version still 3.0.0-SNAPSHOT in branch-2.8. It should be 2.8.0-SNAPSHOT. So, dependency download will fail. After making change to 2.8.0-SNAPSHOT, it passes me locally.

          Show
          rohithsharma Rohith Sharma K S added a comment - I found the issue, failure is because newly added project pom.xml version still 3.0.0-SNAPSHOT in branch-2.8. It should be 2.8.0-SNAPSHOT. So, dependency download will fail. After making change to 2.8.0-SNAPSHOT, it passes me locally.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          The same issue is there in branch-2 also.

          Show
          rohithsharma Rohith Sharma K S added a comment - The same issue is there in branch-2 also.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updating the addendum patch fixing compilation issue in branch-2/branch-2.8

          Show
          rohithsharma Rohith Sharma K S added a comment - Updating the addendum patch fixing compilation issue in branch-2/branch-2.8
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Committed addendum patch to branch-2 and branch-2.8 respectively to unblock compilation issue asap..

          Show
          rohithsharma Rohith Sharma K S added a comment - Committed addendum patch to branch-2 and branch-2.8 respectively to unblock compilation issue asap..
          Hide
          djp Junping Du added a comment -

          It is weird that my build on branch-2/branch-2.8 locally get passed before check in... may due to local cache from build against trunk before? Anyway, thanks Karthik Kambatla and Rohith Sharma K S to report the issue and fix it.

          Show
          djp Junping Du added a comment - It is weird that my build on branch-2/branch-2.8 locally get passed before check in... may due to local cache from build against trunk before? Anyway, thanks Karthik Kambatla and Rohith Sharma K S to report the issue and fix it.

            People

            • Assignee:
              gtCarrera9 Li Lu
              Reporter:
              gtCarrera9 Li Lu
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development