Hadoop Common
  1. Hadoop Common
  2. HADOOP-5002

2 core tests TestFileOutputFormat and TestHarFileSystem are failing in branch 19

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.1
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      tests fail on my RHEL desktop in branch 19

    • Hadoop Flags:
      Reviewed
    • Release Note:
      This patch solves the null pointer exception issue in the 2 core tests TestFileOutputFormat and TestHarFileSystem in branch 19.

      Description

      I see 2 of the core tests(TestHarFileSystem and TestFileOutputFormat) failing in branch 19. These are passing in branch 20 and in trunk.
      I see nullpointer exceptions for the attempts of reducde tasks in the logs(for example, see the following):

      2009-01-09 13:53:34,112 INFO mapred.TaskInProgress (TaskInProgress.java:updateStatus(484)) - Error from attempt_200901091353_0001_r_000000_0: java.lang.NullPointerException
      at org.apache.hadoop.fs.Path.<init>(Path.java:61)
      at org.apache.hadoop.fs.Path.<init>(Path.java:50)
      at org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(HadoopArchives.java:552)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:337)
      at org.apache.hadoop.mapred.Child.main(Child.java:155)

      1. patch-5002.txt
        0.9 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Hide
        Amareshwari Sriramadasu added a comment -

        With HADOOP-4847, setting of work output path is moved to method, initialize(), which is the first method called from run() of Map/Reduce Task, after starting communication thread.
        In branch 0.19, ReduceTask.run() does instantiation of class Reducer in the beginning of the method. i.e. before initialize() call, which is not necessary.
        The patch moves the instantiation of reducer down, where the reducer is actually called i.e. after shuffle and sort.

        Show
        Amareshwari Sriramadasu added a comment - With HADOOP-4847 , setting of work output path is moved to method, initialize(), which is the first method called from run() of Map/Reduce Task, after starting communication thread. In branch 0.19, ReduceTask.run() does instantiation of class Reducer in the beginning of the method. i.e. before initialize() call, which is not necessary. The patch moves the instantiation of reducer down, where the reducer is actually called i.e. after shuffle and sort.
        Hide
        Amareshwari Sriramadasu added a comment -

        This is due to HADOOP-4847

        Show
        Amareshwari Sriramadasu added a comment - This is due to HADOOP-4847
        Hide
        Ravi Gummadi added a comment -

        Patch looks fine.

        Patch testing gave

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no tests are needed for this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

        As this patch makes the 2 tests pass, I guess this doesn't need extra testcases to be written.

        Unit tests also passed on my local machine.

        Show
        Ravi Gummadi added a comment - Patch looks fine. Patch testing gave [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. As this patch makes the 2 tests pass, I guess this doesn't need extra testcases to be written. Unit tests also passed on my local machine.
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching the patch with fix

        Show
        Amareshwari Sriramadasu added a comment - Attaching the patch with fix

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Ravi Gummadi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development