Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1086

hadoop commands in streaming tasks are trying to write to tasktracker's log

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      This patch makes TT to set HADOOP_ROOT_LOGGER to INFO,TLA by default in the environment of taskjvm and its children.

      Description

      As HADOOP_ROOT_LOGGER is not set in the environment by TT for the children, the children of task jvm(in case of streaming) are trying to write to TT's log and getting the following Exception. Jobs are succeeded, but the issue is to be resolved by setting the environment variables by TT for use by children of task jvm in case of streaming job.

      When streaming calls hadoop commands, it's trying to write to TaskTracker log file.

      log4j:ERROR setFile(null,true) call failed.
      java.io.FileNotFoundException:
      /a/b/tasktracker.log (Permission denied)
      at java.io.FileOutputStream.openAppend(Native Method)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
      at org.apache.log4j.FileAppender.setFile(FileAppender.java:290)
      at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164)
      at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216)
      at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
      at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
      at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
      at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
      at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
      at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
      at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
      at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
      at org.apache.log4j.Logger.getLogger(Logger.java:105)
      at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
      at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
      at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
      at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
      at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
      at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:138)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:1880)
      log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].

      1. MR-1086-yhadoop20.patch
        8 kB
        Ravi Gummadi
      2. MR-1086.v1.2.patch
        9 kB
        Ravi Gummadi
      3. MR-1086.v1.1.patch
        8 kB
        Ravi Gummadi
      4. MR-1086.v1.patch
        8 kB
        Ravi Gummadi
      5. MR-1086.patch
        8 kB
        Ravi Gummadi

        Activity

        Hide
        Ravi Gummadi added a comment -

        This issue is only with LinuxTaskController.

        Show
        Ravi Gummadi added a comment - This issue is only with LinuxTaskController.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch for trunk.

        Please review and provide your comments.

        Show
        Ravi Gummadi added a comment - Attaching patch for trunk. Please review and provide your comments.
        Hide
        Hemanth Yamijala added a comment -

        Ravi, this looks mostly good.

        I have very minor comments:

        • I am thinking it makes sense to set the log parameters in HADOOP_CLIENT_OPTS than HADOOP_OPTS. This will make sure they get appended at the end and is more reliable. Further, I think we can assume that the streaming tasks only use the hadoop command line as a client.
        • To be consistent with what is set in the scripts of Hadoop, I would recommend that the value of HADOOP_CLIENT_OPTS is enclosed in double quotes than single.
        • We really don't need to convert taskId or log file size to Strings, as Java will do the conversion itself.
        • In the test case, I suppose we can be a little more strict in verification. Can we check that the env contains 'INFO,TLA' as that's what we are setting. Likewise, we can also verify the log file size is set to a known value by setting it in the corresponding property in the Configuration.
        Show
        Hemanth Yamijala added a comment - Ravi, this looks mostly good. I have very minor comments: I am thinking it makes sense to set the log parameters in HADOOP_CLIENT_OPTS than HADOOP_OPTS. This will make sure they get appended at the end and is more reliable. Further, I think we can assume that the streaming tasks only use the hadoop command line as a client. To be consistent with what is set in the scripts of Hadoop, I would recommend that the value of HADOOP_CLIENT_OPTS is enclosed in double quotes than single. We really don't need to convert taskId or log file size to Strings, as Java will do the conversion itself. In the test case, I suppose we can be a little more strict in verification. Can we check that the env contains 'INFO,TLA' as that's what we are setting. Likewise, we can also verify the log file size is set to a known value by setting it in the corresponding property in the Configuration.
        Hide
        Ravi Gummadi added a comment -

        Attaching new patch with the suggested changes.

        Show
        Ravi Gummadi added a comment - Attaching new patch with the suggested changes.
        Hide
        Hemanth Yamijala added a comment -

        Seems fine. Though not really related to this patch, the place where the localFS is being initialized can avoid a catch and ignore of the IOException, as it is anyway going to die with a NullPointerException later. Since this is indeed an exceptional condition, I would rather we throw the IOException. Looking at code paths, that actually seems to be handled better in the child tasks than NullPointerException is.

        Also can you update the documentation of the getVMEnvironment API to include the new parameters ?

        Other than these changes +1. Please make the new patch with these changes Patch Available.

        Show
        Hemanth Yamijala added a comment - Seems fine. Though not really related to this patch, the place where the localFS is being initialized can avoid a catch and ignore of the IOException, as it is anyway going to die with a NullPointerException later. Since this is indeed an exceptional condition, I would rather we throw the IOException. Looking at code paths, that actually seems to be handled better in the child tasks than NullPointerException is. Also can you update the documentation of the getVMEnvironment API to include the new parameters ? Other than these changes +1. Please make the new patch with these changes Patch Available.
        Hide
        Ravi Gummadi added a comment -

        Attaching new patch with the suggested changes.

        Show
        Ravi Gummadi added a comment - Attaching new patch with the suggested changes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12422326/MR-1086.v1.1.patch
        against trunk revision 825469.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422326/MR-1086.v1.1.patch against trunk revision 825469. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/176/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        Canceling patch to fix the test failure. The test passes locally on my machine as well. But I suspect the reason it is failing on Hudson may be due to the fact that the streaming commands are written without specifying a shell. Naturally, it would be useful to add some log statements, at a minimum to print the output of the streaming job.

        Another issue could be that we are using getBytes() to convert from String to bytes and writing the command to a file. Why not use the more standard BufferedWriter ?

        Show
        Hemanth Yamijala added a comment - Canceling patch to fix the test failure. The test passes locally on my machine as well. But I suspect the reason it is failing on Hudson may be due to the fact that the streaming commands are written without specifying a shell. Naturally, it would be useful to add some log statements, at a minimum to print the output of the streaming job. Another issue could be that we are using getBytes() to convert from String to bytes and writing the command to a file. Why not use the more standard BufferedWriter ?
        Hide
        Ravi Gummadi added a comment -

        Test failure on Hudson was because of the shell specific statement " cat >& /dev/null ". Modified it to " cat > /dev/null 2>&1" in this new patch.

        Show
        Ravi Gummadi added a comment - Test failure on Hudson was because of the shell specific statement " cat >& /dev/null ". Modified it to " cat > /dev/null 2>&1" in this new patch.
        Hide
        Ravi Gummadi added a comment -

        Letting the new patch go through Hudson...

        Show
        Ravi Gummadi added a comment - Letting the new patch go through Hudson...
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12422569/MR-1086.v1.2.patch
        against trunk revision 826635.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422569/MR-1086.v1.2.patch against trunk revision 826635. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/186/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        The changes are fine. Appears my diagnosis for the test failure was not correct. Thanks for fixing the correct bug.

        +1.

        Show
        Hemanth Yamijala added a comment - The changes are fine. Appears my diagnosis for the test failure was not correct. Thanks for fixing the correct bug. +1.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk and branch 0.21. Thanks, Ravi !

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.21. Thanks, Ravi !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #87 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/87/)
        . Setup Hadoop logging environment for tasks to point to task related parameters. Contributed by Ravi Gummadi.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #87 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/87/ ) . Setup Hadoop logging environment for tasks to point to task related parameters. Contributed by Ravi Gummadi.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch for yahoo-hadoop-0.20 branch.
        ant test and ant test-patch passed on my local machine.

        Show
        Ravi Gummadi added a comment - Attaching patch for yahoo-hadoop-0.20 branch. ant test and ant test-patch passed on my local machine.
        Hide
        Hemanth Yamijala added a comment -

        Verified the 20 patch. Looks fine.

        Show
        Hemanth Yamijala added a comment - Verified the 20 patch. Looks fine.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #118 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/118/)
        . Setup Hadoop logging environment for tasks to point to task related parameters. Contributed by Ravi Gummadi.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #118 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/118/ ) . Setup Hadoop logging environment for tasks to point to task related parameters. Contributed by Ravi Gummadi.

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Ravi Gummadi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development