Hadoop Common
  1. Hadoop Common
  2. HADOOP-6757

NullPointerException for hadoop clients launched from streaming tasks

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: scripts
    • Labels:
      None

      Description

      TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE.

      1. BZ-3620565-v1.0.patch
        0.9 kB
        Amar Kamat
      2. HADOOP-6757-v1.0.patch
        1 kB
        Amar Kamat

        Activity

        Hide
        Amar Kamat added a comment -

        Attaching a patch for Yahoo!'s distribution if Hadoop not to be committed here. This patch adds HADOOP_CLIENT_OPTS to HADOOP_OPTS so that the TLA level env settings gets passed correctly.

        Show
        Amar Kamat added a comment - Attaching a patch for Yahoo!'s distribution if Hadoop not to be committed here. This patch adds HADOOP_CLIENT_OPTS to HADOOP_OPTS so that the TLA level env settings gets passed correctly.
        Hide
        Amar Kamat added a comment -

        Attaching a patch for trunk. This patch adds the HADOOP_CLIENT_OPTS to HADOOP_OPTS for the following commands

        • bin/mapred job
        • bin/mapred queue
        • bin/hadoop jar
        Show
        Amar Kamat added a comment - Attaching a patch for trunk. This patch adds the HADOOP_CLIENT_OPTS to HADOOP_OPTS for the following commands bin/mapred job bin/mapred queue bin/hadoop jar
        Hide
        Amar Kamat added a comment -

        Running through Hudson.

        Show
        Amar Kamat added a comment - Running through Hudson.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12443943/HADOOP-6757-v1.0.patch
        against trunk revision 941662.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443943/HADOOP-6757-v1.0.patch against trunk revision 941662. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch looks fine to me.

        Show
        Amareshwari Sriramadasu added a comment - Patch looks fine to me.
        Hide
        Sharad Agarwal added a comment -

        Can we have a testcase for this or the manual steps which were used to verify this ?

        Show
        Sharad Agarwal added a comment - Can we have a testcase for this or the manual steps which were used to verify this ?
        Hide
        Tom White added a comment -

        Marking as open while Sharad's comment is addressed.

        Show
        Tom White added a comment - Marking as open while Sharad's comment is addressed.
        Hide
        Tom White added a comment -

        I think this may be fixed by MAPREDUCE-2372.

        Show
        Tom White added a comment - I think this may be fixed by MAPREDUCE-2372 .
        Hide
        Eli Collins added a comment -

        I think this is fixed in branch-1, and is not relevant to other releases.

        Show
        Eli Collins added a comment - I think this is fixed in branch-1, and is not relevant to other releases.

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development