Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6757

NullPointerException for hadoop clients launched from streaming tasks

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: scripts
        • Labels:
          None

          Description

          TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE.

          1. Text File
            BZ-3620565-v1.0.patch
            0.9 kB
            Amar Kamat
          2. Text File
            HADOOP-6757-v1.0.patch
            1 kB
            Amar Kamat

            Activity

            Descending order - Click to sort in ascending order
            Hide
            Permalink
            eli2 Eli Collins added a comment -

            I think this is fixed in branch-1, and is not relevant to other releases.

            Show
            eli2 Eli Collins added a comment - I think this is fixed in branch-1, and is not relevant to other releases.
            Hide
            Permalink
            tomwhite Tom White added a comment -

            I think this may be fixed by MAPREDUCE-2372.

            Show
            tomwhite Tom White added a comment - I think this may be fixed by MAPREDUCE-2372 .
            Hide
            Permalink
            tomwhite Tom White added a comment -

            Marking as open while Sharad's comment is addressed.

            Show
            tomwhite Tom White added a comment - Marking as open while Sharad's comment is addressed.
            Hide
            Permalink
            sharadag Sharad Agarwal added a comment -

            Can we have a testcase for this or the manual steps which were used to verify this ?

            Show
            sharadag Sharad Agarwal added a comment - Can we have a testcase for this or the manual steps which were used to verify this ?
            Hide
            Permalink
            amareshwari Amareshwari Sriramadasu added a comment -

            Patch looks fine to me.

            Show
            amareshwari Amareshwari Sriramadasu added a comment - Patch looks fine to me.
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12443943/HADOOP-6757-v1.0.patch
            against trunk revision 941662.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443943/HADOOP-6757-v1.0.patch against trunk revision 941662. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/65/console This message is automatically generated.
            Hide
            Permalink
            amar_kamat Amar Kamat added a comment -

            Running through Hudson.

            Show
            amar_kamat Amar Kamat added a comment - Running through Hudson.
            Hide
            Permalink
            amar_kamat Amar Kamat added a comment -

            Attaching a patch for trunk. This patch adds the HADOOP_CLIENT_OPTS to HADOOP_OPTS for the following commands

            • bin/mapred job
            • bin/mapred queue
            • bin/hadoop jar
            Show
            amar_kamat Amar Kamat added a comment - Attaching a patch for trunk. This patch adds the HADOOP_CLIENT_OPTS to HADOOP_OPTS for the following commands bin/mapred job bin/mapred queue bin/hadoop jar
            Hide
            Permalink
            amar_kamat Amar Kamat added a comment -

            Attaching a patch for Yahoo!'s distribution if Hadoop not to be committed here. This patch adds HADOOP_CLIENT_OPTS to HADOOP_OPTS so that the TLA level env settings gets passed correctly.

            Show
            amar_kamat Amar Kamat added a comment - Attaching a patch for Yahoo!'s distribution if Hadoop not to be committed here. This patch adds HADOOP_CLIENT_OPTS to HADOOP_OPTS so that the TLA level env settings gets passed correctly.

              People

              • Assignee:
                amar_kamat Amar Kamat
                Reporter:
                amar_kamat Amar Kamat
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development