Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4006

history server container log web UI sometimes combines stderr/stdout/syslog contents together

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 0.23.2
    • Component/s: jobhistoryserver, mrv2
    • Labels:
      None

      Description

      When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI.

      1. MR4006.txt
        7 kB
        Siddharth Seth

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1022 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1022/)
        MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1022 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1022/ ) MAPREDUCE-4006 . History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/228/)
        merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/228/ ) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #200 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/200/)
        merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #200 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/200/ ) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #987 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/987/)
        MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #987 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/987/ ) MAPREDUCE-4006 . History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #705 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/705/)
        merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734)

        Result = ABORTED
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #705 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/705/ ) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1901 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1901/)
        MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731)

        Result = ABORTED
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1901 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1901/ ) MAPREDUCE-4006 . History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #697 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/697/)
        merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #697 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/697/ ) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #688 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/688/)
        merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #688 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/688/ ) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Thomas Graves added a comment -

        committed to trunk, branch-0.23, and branch-0.23.2.

        Show
        Thomas Graves added a comment - committed to trunk, branch-0.23, and branch-0.23.2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1893 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1893/)
        MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1893 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1893/ ) MAPREDUCE-4006 . History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1967 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1967/)
        MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1967 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1967/ ) MAPREDUCE-4006 . History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
        Hide
        Thomas Graves added a comment -

        +1. Thanks Sid! I'll commit this shortly.

        Show
        Thomas Graves added a comment - +1. Thanks Sid! I'll commit this shortly.
        Hide
        Jason Lowe added a comment -

        +1 lgtm. Verified test fails without change and passes with change.

        Show
        Jason Lowe added a comment - +1 lgtm. Verified test fails without change and passes with change.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12518268/MR4006.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2044//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2044//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518268/MR4006.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2044//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2044//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        Trivial patch. Was appending the entire buffer to the output instead of the amount read.

        Show
        Siddharth Seth added a comment - Trivial patch. Was appending the entire buffer to the output instead of the amount read.

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development