2 core tests TestFileOutputFormat and TestHarFileSystem are failing in branch 19

• Key: HADOOP-5002
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Amareshwari Sriramadasu
• Reporter: Ravi Gummadi
• Affects Version/s: 0.19.1
• Fix Version/s: 0.19.1

Description

I see 2 of the core tests(TestHarFileSystem and TestFileOutputFormat) failing in branch 19. These are passing in branch 20 and in trunk.
I see nullpointer exceptions for the attempts of reducde tasks in the logs(for example, see the following):

2009-01-09 13:53:34,112 INFO mapred.TaskInProgress (TaskInProgress.java:updateStatus(484)) - Error from attempt_200901091353_0001_r_000000_0: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.<init>(Path.java:61)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(HadoopArchives.java:552)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:337)
at org.apache.hadoop.mapred.Child.main(Child.java:155)

Comments

[Amareshwari Sriramadasu]

Attaching the patch with fix

[Ravi Gummadi]

Patch looks fine.

Patch testing gave

[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

As this patch makes the 2 tests pass, I guess this doesn't need extra testcases to be written.

Unit tests also passed on my local machine.

[Amareshwari Sriramadasu]

This is due to HADOOP-4847

[Amareshwari Sriramadasu]

With HADOOP-4847, setting of work output path is moved to method, initialize(), which is the first method called from run() of Map/Reduce Task, after starting communication thread.
In branch 0.19, ReduceTask.run() does instantiation of class Reducer in the beginning of the method. i.e. before initialize() call, which is not necessary.
The patch moves the instantiation of reducer down, where the reducer is actually called i.e. after shuffle and sort.

[Devaraj Das]

I just committed this. Thanks, Amareshwari!