[MAPREDUCE-438] When connecting to HDFS using an IP Task MapRed gets confused when checking the output path.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved

Environment

AIX 6.1 and Solaris

• Assignee: Unassigned Assign to me
• Reporter: Bill Habermaas
• Created: 01/Apr/09 19:05
• Updated: 20/Jul/09 21:07

Description

The exception appearing below happens if you are using an IP address to reference a HDFS file system path. Although the URI authority for either path (whether referenced by hostname:port or IP-address:port) refer to the same filesystem, Task throws an exception because the stringified comparisons of the authority in the URIs are not equal. The FileSystem holds the network location of the namenode as a host name regardless as to whether an IP address or hostname is used when the file system is connected (refer to HADOOP-5191 for an earlier fix specifically for HDFS only).

2009-03-27 04:15:45,045 WARN [Thread-145] org.apache.hadoop.mapred.LocalJobRunner: job_local_0002
java.io.IOException: Can not get the relative path: base = hdfs://10.120.16.68:9000/mydata/2009/03/27/0bab100a-1bf1-499a-935d-bc4b4e94f44c/_temporary/_attempt_local_0002_r_000000_0 child = hdfs://p520aix61.mydomain.com:9000/mydata/2009/03/27/0bab100a-1bf1-499a-935d-bc4b4e94f44c/_temporary/_attempt_local_0002_r_000000_0/part-00000
at org.apache.hadoop.mapred.Task.getFinalPath(Task.java:586)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:599)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:617)
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:561)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:202)

Issue Links

• is related to: HADOOP-5191 After creation and startup of the hadoop namenode on AIX or Solaris, you will only be allowed to connect to the namenode via hostname but not IP.