Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-6415 added a tool to archive aggregated logs into HAR files. It seeds the initial list of applications to process based on apps which have finished aggregated, according to the RM. However, the RM doesn't remember completed applications forever (e.g. failover), so it's possible for the tool to miss applications if they're no longer in the RM.

      Instead, we should do the following:

      1. Seed the initial list of apps based on the aggregated log directories
      2. Make the RM not consider applications "complete" until their log aggregation has reached a terminal state (i.e. DISABLED, SUCCEEDED, FAILED, TIME_OUT).

      #2 will allow #1 to assume that any apps not found in the RM are done aggregating. #1 on it's own should cover most cases though

      1. MAPREDUCE-6480.001.patch
        26 kB
        Robert Kanter
      2. MAPREDUCE-6480.002.patch
        30 kB
        Robert Kanter
      3. MAPREDUCE-6480.003.patch
        37 kB
        Robert Kanter

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #418 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/418/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/pom.xml
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #418 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/418/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/pom.xml hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2358 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2358/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-archive-logs/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2358 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2358/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-archive-logs/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2385/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-tools/hadoop-archive-logs/pom.xml
          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2385/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-tools/hadoop-archive-logs/pom.xml hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1180 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1180/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1180 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1180/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #441 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/441/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/pom.xml
          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #441 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/441/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/pom.xml hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8520 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8520/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8520 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8520/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #447 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/447/)
          MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073)

          • hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-archive-logs/pom.xml
          • hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
          • hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #447 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/447/ ) MAPREDUCE-6480 . archive-logs tool may miss applications (rkanter) (rkanter: rev d3c49e76624b7e42a1321c649a1d7bb9906b3073) hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-archive-logs/pom.xml hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java hadoop-tools/hadoop-archive-logs/dev-support/findbugs-exclude.xml
          Hide
          rkanter Robert Kanter added a comment -

          Thanks for the review Anubhav Dhoot. Committed to trunk and branch-2!

          Show
          rkanter Robert Kanter added a comment - Thanks for the review Anubhav Dhoot . Committed to trunk and branch-2!
          Hide
          adhoot Anubhav Dhoot added a comment -

          +1

          Show
          adhoot Anubhav Dhoot added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 23s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 0s There were no new javac warning messages.
          +1 javadoc 10m 28s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 19s The applied patch generated 1 new checkstyle issues (total was 4, now 4).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings, and fixes 3 pre-existing warnings.
          +1 tools/hadoop tests 0m 33s Tests passed in hadoop-archive-logs.
              39m 58s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762431/MAPREDUCE-6480.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e65c5
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt
          hadoop-archive-logs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/artifact/patchprocess/testrun_hadoop-archive-logs.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 23s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 28s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 19s The applied patch generated 1 new checkstyle issues (total was 4, now 4). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings, and fixes 3 pre-existing warnings. +1 tools/hadoop tests 0m 33s Tests passed in hadoop-archive-logs.     39m 58s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762431/MAPREDUCE-6480.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt hadoop-archive-logs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/artifact/patchprocess/testrun_hadoop-archive-logs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6019/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          003 patch addresses Anubhav's comments:

          1. Ya. For FAILED, we don't know why, so I figured it was safest to just not do anything to the logs. For TIMED_OUT, it seemed like the idea is that nothing bad happened, but we're not expecting any more logs.

          2. Done

          3. This was generated by IntelliJ. I'd rather leave it in as-is to be safe.

          4. I've added a -verbose option, which prints out a lot more details about what's happening.

          5. Done

          6. Done

          7. Done

          8. I think we should do both. You're right that we could break out if there's an error with just one app. At the same time, if the user running the tool doesn't have permission to access the logs for a specific user, we don't want to keep trying over and over again.

          8 (#2). Done

          9. Done

          Show
          rkanter Robert Kanter added a comment - 003 patch addresses Anubhav's comments: 1. Ya. For FAILED, we don't know why, so I figured it was safest to just not do anything to the logs. For TIMED_OUT, it seemed like the idea is that nothing bad happened, but we're not expecting any more logs. 2. Done 3. This was generated by IntelliJ. I'd rather leave it in as-is to be safe. 4. I've added a -verbose option, which prints out a lot more details about what's happening. 5. Done 6. Done 7. Done 8. I think we should do both. You're right that we could break out if there's an error with just one app. At the same time, if the user running the tool doesn't have permission to access the logs for a specific user, we don't want to keep trying over and over again. 8 (#2). Done 9. Done
          Hide
          adhoot Anubhav Dhoot added a comment -

          Robert Kanter the overall approach looks good.
          Some questions and comments.

          Re checkAggregatedStatus and various LogAggregationStatus states
          1. a. Looks like TIMED_OUT state will be considered eligible for logaggregation. That seems the right thing to do
          1. b. Similarly FAILED is not considered for log aggregation while earlier it was, right? Should it also be considered for aggregation? Looking at the description it indicates some log has error in aggreation. Are we not considering it because it may cause some logs to be left behind? If thats the reason I am ok with the current patch approach. Just want to confirm.

          2. If a new non terminal LogAggregationStatus gets added to the enum we will not filter it out in checkAggregatedStatus. Is that a problem that we accidentally process it? If so we should extract a function and add a unit test case to catch those newly added states and explicitly fail the unit test.

          3. consider reversing the conditions for the user != null in AppInfo#equals to make it easier to read

          return user != null ? user.equals(appInfo.user) : appInfo.user == null;
          

          4. consider adding log lines when we skip an application because the maxTotalSize is exceeded and also when we get an IOException.
          5. FileWriter import is now unused
          6. consider seedAndCheckFiles -> checkFilesAndSeedApps
          7. consider checkAggregatedStatus -> filterAppsByAggregatedStatus

          8. In seedAndCheckFiles we have a catch for the outer iteration. This could potentially break out of the loop on first app with an error and not process any more apps for that user. Should it instead be like the old way where we wrap the innermost

           FileStatus[] files = fs.listStatus(appLogPath); 

          That will also match the comment

           // Ignore any apps we can't read 

          TestHadoopArchiveLogs
          8. Consider extracting out System.getProperty("user.name") into a variable to avoid repeated calls

          nits:
          9. Formatting inconsistency in in the findbugs exclude

          Show
          adhoot Anubhav Dhoot added a comment - Robert Kanter the overall approach looks good. Some questions and comments. Re checkAggregatedStatus and various LogAggregationStatus states 1. a. Looks like TIMED_OUT state will be considered eligible for logaggregation. That seems the right thing to do 1. b. Similarly FAILED is not considered for log aggregation while earlier it was, right? Should it also be considered for aggregation? Looking at the description it indicates some log has error in aggreation. Are we not considering it because it may cause some logs to be left behind? If thats the reason I am ok with the current patch approach. Just want to confirm. 2. If a new non terminal LogAggregationStatus gets added to the enum we will not filter it out in checkAggregatedStatus. Is that a problem that we accidentally process it? If so we should extract a function and add a unit test case to catch those newly added states and explicitly fail the unit test. 3. consider reversing the conditions for the user != null in AppInfo#equals to make it easier to read return user != null ? user.equals(appInfo.user) : appInfo.user == null; 4. consider adding log lines when we skip an application because the maxTotalSize is exceeded and also when we get an IOException. 5. FileWriter import is now unused 6. consider seedAndCheckFiles -> checkFilesAndSeedApps 7. consider checkAggregatedStatus -> filterAppsByAggregatedStatus 8. In seedAndCheckFiles we have a catch for the outer iteration. This could potentially break out of the loop on first app with an error and not process any more apps for that user. Should it instead be like the old way where we wrap the innermost FileStatus[] files = fs.listStatus(appLogPath); That will also match the comment // Ignore any apps we can't read TestHadoopArchiveLogs 8. Consider extracting out System.getProperty("user.name") into a variable to avoid repeated calls nits: 9. Formatting inconsistency in in the findbugs exclude
          Hide
          rkanter Robert Kanter added a comment -

          One of the checkstyle warnings is that finishTime hides a field here:

          public void setFinishTime(long finishTime) {
            this.finishTime = finishTime;
          }
          

          but this is pretty standard practice for a setter, isn't it?

          Show
          rkanter Robert Kanter added a comment - One of the checkstyle warnings is that finishTime hides a field here: public void setFinishTime( long finishTime) { this .finishTime = finishTime; } but this is pretty standard practice for a setter, isn't it?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 36s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 15s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 21s The applied patch generated 2 new checkstyle issues (total was 5, now 6).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 0m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings, and fixes 3 pre-existing warnings.
          +1 tools/hadoop tests 0m 32s Tests passed in hadoop-archive-logs.
              39m 5s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12757174/MAPREDUCE-6480.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6b97fa6
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt
          hadoop-archive-logs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/artifact/patchprocess/testrun_hadoop-archive-logs.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 36s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 21s The applied patch generated 2 new checkstyle issues (total was 5, now 6). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 0m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings, and fixes 3 pre-existing warnings. +1 tools/hadoop tests 0m 32s Tests passed in hadoop-archive-logs.     39m 5s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12757174/MAPREDUCE-6480.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6b97fa6 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt hadoop-archive-logs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/artifact/patchprocess/testrun_hadoop-archive-logs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6002/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          The 002 patch fixes the failing tests, checkstyle, and findbugs.

          Show
          rkanter Robert Kanter added a comment - The 002 patch fixes the failing tests, checkstyle, and findbugs.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 42s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 46s There were no new javac warning messages.
          +1 javadoc 11m 17s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 21s The applied patch generated 7 new checkstyle issues (total was 4, now 11).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 0m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 tools/hadoop tests 0m 30s Tests failed in hadoop-archive-logs.
              42m 12s  



          Reason Tests
          Failed unit tests hadoop.tools.TestHadoopArchiveLogsRunner
            hadoop.tools.TestHadoopArchiveLogs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756389/MAPREDUCE-6480.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0832b38
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt
          hadoop-archive-logs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/testrun_hadoop-archive-logs.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 42s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 46s There were no new javac warning messages. +1 javadoc 11m 17s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 21s The applied patch generated 7 new checkstyle issues (total was 4, now 11). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 0m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 tools/hadoop tests 0m 30s Tests failed in hadoop-archive-logs.     42m 12s   Reason Tests Failed unit tests hadoop.tools.TestHadoopArchiveLogsRunner   hadoop.tools.TestHadoopArchiveLogs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756389/MAPREDUCE-6480.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0832b38 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt hadoop-archive-logs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/testrun_hadoop-archive-logs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          The patch takes care of #1 by refactoring a bunch of code to seed the list from the aggregated log files and filter out any that haven't finished aggregating according to the RM.

          We can do #2 in a followup YARN JIRA.

          Show
          rkanter Robert Kanter added a comment - The patch takes care of #1 by refactoring a bunch of code to seed the list from the aggregated log files and filter out any that haven't finished aggregating according to the RM. We can do #2 in a followup YARN JIRA.

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development