Hadoop Common
  1. Hadoop Common
  2. HADOOP-5002

2 core tests TestFileOutputFormat and TestHarFileSystem are failing in branch 19

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.1
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      tests fail on my RHEL desktop in branch 19

    • Hadoop Flags:
      Reviewed
    • Release Note:
      This patch solves the null pointer exception issue in the 2 core tests TestFileOutputFormat and TestHarFileSystem in branch 19.

      Description

      I see 2 of the core tests(TestHarFileSystem and TestFileOutputFormat) failing in branch 19. These are passing in branch 20 and in trunk.
      I see nullpointer exceptions for the attempts of reducde tasks in the logs(for example, see the following):

      2009-01-09 13:53:34,112 INFO mapred.TaskInProgress (TaskInProgress.java:updateStatus(484)) - Error from attempt_200901091353_0001_r_000000_0: java.lang.NullPointerException
      at org.apache.hadoop.fs.Path.<init>(Path.java:61)
      at org.apache.hadoop.fs.Path.<init>(Path.java:50)
      at org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(HadoopArchives.java:552)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:337)
      at org.apache.hadoop.mapred.Child.main(Child.java:155)

      1. patch-5002.txt
        0.9 kB
        Amareshwari Sriramadasu

        Activity

        Ravi Gummadi created issue -
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching the patch with fix

        Show
        Amareshwari Sriramadasu added a comment - Attaching the patch with fix
        Amareshwari Sriramadasu made changes -
        Field Original Value New Value
        Attachment patch-5002.txt [ 12397494 ]
        Ravi Gummadi made changes -
        Assignee Amareshwari Sriramadasu [ amareshwari ]
        Hide
        Ravi Gummadi added a comment -

        Patch looks fine.

        Patch testing gave

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no tests are needed for this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

        As this patch makes the 2 tests pass, I guess this doesn't need extra testcases to be written.

        Unit tests also passed on my local machine.

        Show
        Ravi Gummadi added a comment - Patch looks fine. Patch testing gave [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. As this patch makes the 2 tests pass, I guess this doesn't need extra testcases to be written. Unit tests also passed on my local machine.
        Ravi Gummadi made changes -
        Release Note This patch solves the null pointer exception issue in the 2 core tests TestFileOutputFormat and TestHarFileSystem in branch 19.
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        This is due to HADOOP-4847

        Show
        Amareshwari Sriramadasu added a comment - This is due to HADOOP-4847
        Amareshwari Sriramadasu made changes -
        Fix Version/s 0.19.1 [ 12313473 ]
        Affects Version/s 0.19.0 [ 12313211 ]
        Affects Version/s 0.19.1 [ 12313473 ]
        Component/s mapred [ 12310690 ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        With HADOOP-4847, setting of work output path is moved to method, initialize(), which is the first method called from run() of Map/Reduce Task, after starting communication thread.
        In branch 0.19, ReduceTask.run() does instantiation of class Reducer in the beginning of the method. i.e. before initialize() call, which is not necessary.
        The patch moves the instantiation of reducer down, where the reducer is actually called i.e. after shuffle and sort.

        Show
        Amareshwari Sriramadasu added a comment - With HADOOP-4847 , setting of work output path is moved to method, initialize(), which is the first method called from run() of Map/Reduce Task, after starting communication thread. In branch 0.19, ReduceTask.run() does instantiation of class Reducer in the beginning of the method. i.e. before initialize() call, which is not necessary. The patch moves the instantiation of reducer down, where the reducer is actually called i.e. after shuffle and sort.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Devaraj Das made changes -
        Hadoop Flags [Reviewed]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        5h 7m 1 Ravi Gummadi 09/Jan/09 14:20
        Patch Available Patch Available Resolved Resolved
        11d 16h 10m 1 Devaraj Das 21/Jan/09 06:31
        Resolved Resolved Closed Closed
        34d 17h 6m 1 Nigel Daley 24/Feb/09 23:37

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Ravi Gummadi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development