Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6875 New aggregated log file format for YARN log aggregation.
  3. YARN-7194

Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. YARN-7194.1.patch
      1 kB
      Xuan Gong
    2. YARN-7194.2.patch
      1 kB
      Xuan Gong
    3. YARN-7194.3.patch
      3 kB
      Xuan Gong

      Activity

      Hide
      xgong Xuan Gong added a comment -

      It happens when we are not in partial log aggregation.
      When we are not in partial log aggregation, we will not create intermediate checksum file, so, when we do postWrite, we would call

      deleteFileWithRetries(fc, ugi, remoteLogCheckSumFile);
      

      which will throw out exception, and complain that the remoteLogCheckSumFile can not be found. So, we would mark the log aggregation status as Failed when we catch the exception.

      Show
      xgong Xuan Gong added a comment - It happens when we are not in partial log aggregation. When we are not in partial log aggregation, we will not create intermediate checksum file, so, when we do postWrite, we would call deleteFileWithRetries(fc, ugi, remoteLogCheckSumFile); which will throw out exception, and complain that the remoteLogCheckSumFile can not be found. So, we would mark the log aggregation status as Failed when we catch the exception.
      Hide
      xgong Xuan Gong added a comment -

      Uploaded a simple fix:
      When we are not in partial log aggregation, we would not try to delete the checksum file in postWrite

      Show
      xgong Xuan Gong added a comment - Uploaded a simple fix: When we are not in partial log aggregation, we would not try to delete the checksum file in postWrite
      Hide
      xgong Xuan Gong added a comment -

      Trivial fix without testcases

      Show
      xgong Xuan Gong added a comment - Trivial fix without testcases
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 12s Docker mode activated.
            Prechecks
      +1 @author 0m 0s The patch does not contain any @author tags.
      -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
            trunk Compile Tests
      +1 mvninstall 14m 40s trunk passed
      +1 compile 0m 35s trunk passed
      +1 checkstyle 0m 24s trunk passed
      +1 mvnsite 0m 38s trunk passed
      +1 findbugs 1m 22s trunk passed
      +1 javadoc 0m 42s trunk passed
            Patch Compile Tests
      +1 mvninstall 0m 37s the patch passed
      +1 compile 0m 37s the patch passed
      +1 javac 0m 37s the patch passed
      +1 checkstyle 0m 23s the patch passed
      +1 mvnsite 0m 39s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      -1 findbugs 1m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
      +1 javadoc 0m 41s the patch passed
            Other Tests
      -1 unit 2m 36s hadoop-yarn-common in the patch failed.
      +1 asflicense 0m 14s The patch does not generate ASF License warnings.
      27m 16s



      Reason Tests
      FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Potentially dangerous use of non-short-circuit logic in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.postWrite(LogAggregationFileControllerContext) At LogAggregationIndexedFileController.java:logic in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.postWrite(LogAggregationFileControllerContext) At LogAggregationIndexedFileController.java:[line 396]
      Failed junit tests hadoop.yarn.logaggregation.filecontroller.ifile.TestLogAggregationIndexFileController



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:71bbb86
      JIRA Issue YARN-7194
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887001/YARN-7194.1.patch
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux 25786756f7d4 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 73aed34
      Default Java 1.8.0_144
      findbugs v3.1.0-RC1
      findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17448/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html
      unit https://builds.apache.org/job/PreCommit-YARN-Build/17448/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
      Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17448/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/17448/console
      Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 14m 40s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 38s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 42s trunk passed       Patch Compile Tests +1 mvninstall 0m 37s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 41s the patch passed       Other Tests -1 unit 2m 36s hadoop-yarn-common in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 27m 16s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common   Potentially dangerous use of non-short-circuit logic in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.postWrite(LogAggregationFileControllerContext) At LogAggregationIndexedFileController.java:logic in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.postWrite(LogAggregationFileControllerContext) At LogAggregationIndexedFileController.java: [line 396] Failed junit tests hadoop.yarn.logaggregation.filecontroller.ifile.TestLogAggregationIndexFileController Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-7194 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887001/YARN-7194.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 25786756f7d4 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 73aed34 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17448/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html unit https://builds.apache.org/job/PreCommit-YARN-Build/17448/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17448/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/17448/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      xgong Xuan Gong added a comment -

      Fix the testcase failure.

      Show
      xgong Xuan Gong added a comment - Fix the testcase failure.
      Hide
      hadoopqa Hadoop QA added a comment -
      +1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 19s Docker mode activated.
            Prechecks
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
            trunk Compile Tests
      +1 mvninstall 13m 47s trunk passed
      +1 compile 0m 32s trunk passed
      +1 checkstyle 0m 23s trunk passed
      +1 mvnsite 0m 34s trunk passed
      +1 findbugs 1m 11s trunk passed
      +1 javadoc 0m 43s trunk passed
            Patch Compile Tests
      +1 mvninstall 0m 39s the patch passed
      +1 compile 0m 31s the patch passed
      +1 javac 0m 31s the patch passed
      +1 checkstyle 0m 22s the patch passed
      +1 mvnsite 0m 40s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 findbugs 1m 32s the patch passed
      +1 javadoc 0m 37s the patch passed
            Other Tests
      +1 unit 2m 49s hadoop-yarn-common in the patch passed.
      +1 asflicense 0m 17s The patch does not generate ASF License warnings.
      26m 17s



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:71bbb86
      JIRA Issue YARN-7194
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887017/YARN-7194.3.patch
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux a64670c63e69 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / e0b3c64
      Default Java 1.8.0_144
      findbugs v3.1.0-RC1
      Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17451/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/17451/console
      Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 47s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 34s trunk passed +1 findbugs 1m 11s trunk passed +1 javadoc 0m 43s trunk passed       Patch Compile Tests +1 mvninstall 0m 39s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests +1 unit 2m 49s hadoop-yarn-common in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 26m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-7194 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887017/YARN-7194.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a64670c63e69 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0b3c64 Default Java 1.8.0_144 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17451/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/17451/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      djp Junping Du added a comment -

      Thanks Xuan Gong for fixing this issue. +1. Committing it in.

      Show
      djp Junping Du added a comment - Thanks Xuan Gong for fixing this issue. +1. Committing it in.
      Hide
      djp Junping Du added a comment -

      I have commit the patch to trunk and branch-2. Thanks Xuan Gong for patch contribution!

      Show
      djp Junping Du added a comment - I have commit the patch to trunk and branch-2. Thanks Xuan Gong for patch contribution!
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12869 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12869/)
      YARN-7194. Log aggregation status is always Failed with the newly added (junping_du: rev c92c1d521eadfd8a4cd8205cc6aee74816f353f4)

      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/LogAggregationIndexedFileController.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/TestLogAggregationIndexFileController.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12869 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12869/ ) YARN-7194 . Log aggregation status is always Failed with the newly added (junping_du: rev c92c1d521eadfd8a4cd8205cc6aee74816f353f4) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/LogAggregationIndexedFileController.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/TestLogAggregationIndexFileController.java

        People

        • Assignee:
          xgong Xuan Gong
          Reporter:
          xgong Xuan Gong
        • Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development