Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2754

MR-279: AM logs are incorrectly going to stderr and error messages going incorrectly to stdout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The log messages for AM container are going into stderr instead of syslog. Also, stderr and stdout roles are reversed.

      1. MAPREDUCE-2754.3.patch
        3 kB
        Ravi Teja Ch N V
      2. MAPREDUCE-2754.2.patch
        2 kB
        Ravi Teja Ch N V
      3. MAPREDUCE-2754.1.patch
        2 kB
        Ravi Teja Ch N V
      4. MAPREDUCE-2754.patch
        1.0 kB
        Ravi Teja Ch N V

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          22h 22m 3 Ravi Teja Ch N V 22/Sep/11 14:15
          Open Open Patch Available Patch Available
          53d 19h 24m 4 Ravi Teja Ch N V 22/Sep/11 14:16
          Patch Available Patch Available Resolved Resolved
          2h 4m 1 Vinod Kumar Vavilapalli 22/Sep/11 16:20
          Resolved Resolved Closed Closed
          53d 9h 28m 1 Arun C Murthy 15/Nov/11 00:48
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #23 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/23/)
          MAPREDUCE-2754. svn merge -c r1174194 --ignore-ancestry ../../trunk/

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174195
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #23 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/23/ ) MAPREDUCE-2754 . svn merge -c r1174194 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174195 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #839 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/839/)
          MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #839 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/839/ ) MAPREDUCE-2754 . Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #809 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/809/)
          MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #809 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/809/ ) MAPREDUCE-2754 . Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #18 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/18/)
          MAPREDUCE-2754. svn merge -c r1174194 --ignore-ancestry ../../trunk/

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174195
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #18 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/18/ ) MAPREDUCE-2754 . svn merge -c r1174194 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174195 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #956 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/956/)
          MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #956 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/956/ ) MAPREDUCE-2754 . Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1016 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1016/)
          MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1016 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1016/ ) MAPREDUCE-2754 . Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #938 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/938/)
          MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #938 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/938/ ) MAPREDUCE-2754 . Fixed MR AM stdout, stderr and syslog to redirect to correct log-files. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174194 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this trunk and branch-0.23. Thanks Ravi Teja!

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this trunk and branch-0.23. Thanks Ravi Teja!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12496105/MAPREDUCE-2754.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/823//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/823//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496105/MAPREDUCE-2754.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/823//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/823//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Can be blamed on me,as I just checked for the log not the content when rebasing the patch.MAPREDUCE-3044 has induced it.The property name was distorted after it got refactored.Fixed it in the patch.

          Actually you need a pat on the back! I was trying to figure this out nearly one hour, but couldn't. Great catch w.r.t the MAPREDUCE-3044 bug!

          Patch looks good. Let's wait for Jenkins to bless the patch before I commit this.

          Show
          Vinod Kumar Vavilapalli added a comment - Can be blamed on me,as I just checked for the log not the content when rebasing the patch. MAPREDUCE-3044 has induced it.The property name was distorted after it got refactored.Fixed it in the patch. Actually you need a pat on the back! I was trying to figure this out nearly one hour, but couldn't. Great catch w.r.t the MAPREDUCE-3044 bug! Patch looks good. Let's wait for Jenkins to bless the patch before I commit this.
          Hide
          Ravi Teja Ch N V added a comment -

          Can be blamed on me,as I just checked for the log not the content when rebasing the patch.MAPREDUCE-3044 has induced it.The property name was distorted after it got refactored.Fixed it in the patch.

          Show
          Ravi Teja Ch N V added a comment - Can be blamed on me,as I just checked for the log not the content when rebasing the patch. MAPREDUCE-3044 has induced it.The property name was distorted after it got refactored.Fixed it in the patch.
          Ravi Teja Ch N V made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ravi Teja Ch N V made changes -
          Attachment MAPREDUCE-2754.3.patch [ 12496105 ]
          Ravi Teja Ch N V made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Ravi Teja Ch N V added a comment -

          Yes Vinod, I have tested on a single node cluster.It was fine.I will check with multi node and revert back. Thanks.

          Show
          Ravi Teja Ch N V added a comment - Yes Vinod, I have tested on a single node cluster.It was fine.I will check with multi node and revert back. Thanks.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Ravi Teja, because writing tests for this isn't easy, I thought I'll run the integration test TestMRJobs and verify manually before committing. But when I ran the test, though syslog was getting created, the logs for AM container aren't there. Can you please take a look?

          Also, you mentioned that you verified the fix, was it on a single node?

          Show
          Vinod Kumar Vavilapalli added a comment - Ravi Teja, because writing tests for this isn't easy, I thought I'll run the integration test TestMRJobs and verify manually before committing. But when I ran the test, though syslog was getting created, the logs for AM container aren't there. Can you please take a look? Also, you mentioned that you verified the fix, was it on a single node?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12496070/MAPREDUCE-2754.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/820//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/820//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496070/MAPREDUCE-2754.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/820//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/820//console This message is automatically generated.
          Hide
          Ravi Teja Ch N V added a comment -

          Hi Vinod, Have rebased and verified the patch on the latest code. Thanks.

          Show
          Ravi Teja Ch N V added a comment - Hi Vinod, Have rebased and verified the patch on the latest code. Thanks.
          Ravi Teja Ch N V made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ravi Teja Ch N V made changes -
          Attachment MAPREDUCE-2754.2.patch [ 12496070 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This patch is completely broken by MAPREDUCE-2880. Ravi Teja, can you please look at the latest code and update this patch? Thanks!

          Show
          Vinod Kumar Vavilapalli added a comment - This patch is completely broken by MAPREDUCE-2880 . Ravi Teja, can you please look at the latest code and update this patch? Thanks!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12495371/MAPREDUCE-2754.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/811//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/811//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495371/MAPREDUCE-2754.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/811//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/811//console This message is automatically generated.
          Ravi Teja Ch N V made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ravi Teja Ch N V made changes -
          Attachment MAPREDUCE-2754.1.patch [ 12495371 ]
          Hide
          Ravi Teja Ch N V added a comment -

          Attached the patch along with the review comments addressed.

          Show
          Ravi Teja Ch N V added a comment - Attached the patch along with the review comments addressed.
          Hide
          Ravi Teja Ch N V added a comment -

          Thanks for the review Vinod...

          You will also need to set corresponding log4j properties

          Since the issue now is with the MRAppmaster log settings, I think we can reuse the container-log4j.properties which is used by the containers, for MRAppMaster also, which internally uses CLA as hadoop.root.logger.

          Show
          Ravi Teja Ch N V added a comment - Thanks for the review Vinod... You will also need to set corresponding log4j properties Since the issue now is with the MRAppmaster log settings, I think we can reuse the container-log4j.properties which is used by the containers, for MRAppMaster also, which internally uses CLA as hadoop.root.logger.
          Vinod Kumar Vavilapalli made changes -
          Summary MR-279: Error messages are being moved to stdout instead of syslog. MR-279: AM logs are incorrectly going to stderr and error messages going incorrectly to stdout
          Vinod Kumar Vavilapalli made changes -
          Link This issue duplicates MAPREDUCE-2755 [ MAPREDUCE-2755 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is part of MAPREDUCE-2755 [ MAPREDUCE-2755 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is part of MAPREDUCE-2755 [ MAPREDUCE-2755 ]
          Vinod Kumar Vavilapalli made changes -
          Summary MR-279: Error messages are being moved to stdout instead of stderr MR-279: Error messages are being moved to stdout instead of syslog.
          Description Error messages flagging the reason for app failures are currently being moved to stdout of container logs instead of stderr. The log messages for AM container are going into stderr instead of syslog. Also, stderr and stdout roles are reversed.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I am fixing the title and description to reflect what we really want.

          Show
          Vinod Kumar Vavilapalli added a comment - I am fixing the title and description to reflect what we really want.
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Ravi Teja, we need the logs to go to syslog like for other containers. This patch is needed, but not sufficient. The logs are going into stdout because hadoop.root.logger is set to console, can you change that to CLA (CLA for ContainerLogAppender). You will also need to set corresponding log4j properties, see the container construction from TaskAttemptImpl.

          Show
          Vinod Kumar Vavilapalli added a comment - Ravi Teja, we need the logs to go to syslog like for other containers. This patch is needed, but not sufficient. The logs are going into stdout because hadoop.root.logger is set to console, can you change that to CLA (CLA for ContainerLogAppender). You will also need to set corresponding log4j properties, see the container construction from TaskAttemptImpl.
          Hide
          Ravi Teja Ch N V added a comment -

          A minor change, checked the patch manually.

          Show
          Ravi Teja Ch N V added a comment - A minor change, checked the patch manually.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12495210/MAPREDUCE-2754.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/798//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/798//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495210/MAPREDUCE-2754.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/798//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/798//console This message is automatically generated.
          Hide
          Ravi Teja Ch N V added a comment -

          The problem is the order of redirection of logs was wrong.

          MapReduceChildJVM.java
          vargs.add("1>" + getTaskLogFile(logDir, TaskLog.LogName.STDERR));
          vargs.add("2>" + getTaskLogFile(logDir, TaskLog.LogName.STDOUT));
          

          It should be 1>STDOUT and 2>STDERR.

          Attached patch for the same.

          Show
          Ravi Teja Ch N V added a comment - The problem is the order of redirection of logs was wrong. MapReduceChildJVM.java vargs.add( "1>" + getTaskLogFile(logDir, TaskLog.LogName.STDERR)); vargs.add( "2>" + getTaskLogFile(logDir, TaskLog.LogName.STDOUT)); It should be 1>STDOUT and 2>STDERR. Attached patch for the same.
          Ravi Teja Ch N V made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ravi Teja Ch N V made changes -
          Attachment MAPREDUCE-2754.patch [ 12495210 ]
          Ravi Teja Ch N V made changes -
          Assignee Ravi Teja Ch N V [ raviteja ]
          Arun C Murthy made changes -
          Field Original Value New Value
          Priority Minor [ 4 ] Blocker [ 1 ]
          Ramya Sunil created issue -

            People

            • Assignee:
              Ravi Teja Ch N V
              Reporter:
              Ramya Sunil
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development