Hadoop YARN
  1. Hadoop YARN
  2. YARN-448

Remove unnecessary hflush from log aggregation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.7, 2.1.0-beta
    • Fix Version/s: 3.0.0, 0.23.7, 2.1.0-beta
    • Component/s: nodemanager
    • Labels:
      None

      Description

      AggregatedLogFormat#writeVersion() calls hflush() after writing the version. Calling hflush does not seem to be necessary. It can add a lot of load to hdfs in a big busy cluster.

        Activity

        Hide
        Robert Joseph Evans added a comment -

        Yes I don't see any real reason to have that hflush there. If log aggregation crashed either before or after that point it really would not make that much of a difference. +1, assuming Jenkins comes back with a +1.

        Show
        Robert Joseph Evans added a comment - Yes I don't see any real reason to have that hflush there. If log aggregation crashed either before or after that point it really would not make that much of a difference. +1, assuming Jenkins comes back with a +1.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12571892/yarn-448.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/458//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/458//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571892/yarn-448.patch.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/458//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/458//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        Test not included since it does not affect normal cases. Even in failure cases, no current error handling in log aggregation is affected by the existence or absence of version record in a log that failed during aggregation.

        Show
        Kihwal Lee added a comment - Test not included since it does not affect normal cases. Even in failure cases, no current error handling in log aggregation is affected by the existence or absence of version record in a log that failed during aggregation.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Kihwal,

        I put this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Kihwal, I put this into trunk, branch-2, and branch-0.23
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3412 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3412/)
        YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452475)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452475
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3412 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3412/ ) YARN-448 . Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452475) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452475 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #146 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/146/)
        YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452475)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452475
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #146 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/146/ ) YARN-448 . Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452475) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452475 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #544 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/544/)
        svn merge -c 1452475 FIXES: YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452478)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #544 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/544/ ) svn merge -c 1452475 FIXES: YARN-448 . Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452478) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452478 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1335 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1335/)
        YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452475)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452475
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1335 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1335/ ) YARN-448 . Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) (Revision 1452475) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452475 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development