Removal of default port# in NameNode.getUri() cause a map/reduce job failed to prompt temporay output

• Key: HADOOP-4717
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Hairong Kuang
• Reporter: Hairong Kuang
• Affects Version/s: 0.18.0
• Fix Version/s: 0.18.3

Description

Problem reported here is that when the default port number (8020) is specified in the output, job succeeds but no output is created. The cause of the problem is that "listStatus" call drops the port number because NameNode.getUri removes the default port#.

Assuming that a map/reduce output directory is set to be "hdfs://localhost:8020/out", A call "listStatus" on any of its sub directory, for example, "hdfs://localhost:8020/out/tempXX", returns results like below:

hdfs://localhost/out/tempXX/part-00005

Because of this, Task.java
574 private Path getFinalPath(Path jobOutputDir, Path taskOutput) {
575 URI relativePath = taskOutputPath.toUri().relativize(taskOutput.toUri());

does not get the correct relativePath because TaskOutputPath contain ports, but taskOutput doesn't.

It seems to me that the problem could be fixed if we make Path.makeQualified() to return the same path not matter the input path contains the default port or not.

Comments

[Doug Cutting]

It seems to me that the output directory should somewhere be normalized by calling FileSystem#makeQualified() on it, so that it's of the form "hdfs://localhost/out/".

[Hairong Kuang]

I wrote a test. It showed that FileSystem#makeQualified() did not remove the default port# even if the input path contains the default port #.

[Doug Cutting]

Here's a patch that changes DistributedFileSystem#makeQualified() to remove the default port if it's specified. Does that fix things for you?

[Hairong Kuang]

Yes, it works as long as we also fix HADOOP-4746. Doug, could you please include a junit test?

[Koji Noguchi]

I understand that HADOOP-4717&HADOOP-4746 would fix the problem, but can we throw an Exception when

575 URI relativePath = taskOutputPath.toUri().relativize(taskOutput.toUri());

doesn't return a relativePath?
If we hit a similar issue again, I would rather have the job fail
than job returning 0 but silently deleting the output.

[Hairong Kuang]

In addition to Doug's change, this patch
1. throws IOException if relativitize fails as Koji suggested;
2. add a unit test to make sure a map/reduce job with output path containing no port works.

[Hairong Kuang]

This new patch makes two changes in my newly added unit test:
1. When dfs cluster fails to start because of server binding exception, log the error and skip the test;
2. The map/reduce job has a output path that includes the default NameNode port#.

[Doug Cutting]

+1 This looks good to me.

[Hairong Kuang]

Ant test-core succeeded:
BUILD SUCCESSFUL
Total time: 113 minutes 54 seconds

Ant test-patch succeeded:
[exec] +1 overall.

[exec] +1 @author. The patch does not contain any @author tags.

[exec] +1 tests included. The patch appears to include 3 new or modified tests.

[exec] +1 javadoc. The javadoc tool did not generate any warning messages.

[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

[Hairong Kuang]

I've just committed this.

[Hudson]

Integrated in Hadoop-trunk #680 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/680/)