[HADOOP-5759] IllegalArgumentException when CombineFileInputFormat is used as job InputFormat

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1, 0.20.205.0
• Fix Version/s: 0.20.2
• Assignee: Amareshwari Sriramadasu
• Reporter: Amareshwari Sriramadasu
• Created: 30/Apr/09 08:07
• Updated: 08/Jun/11 19:15
• Resolved: 27/May/09 04:42

Description

As per my understanding, CombineFileInputFormat is creating splits with rackname as split location.
When I use CombineFileInputFormat as the InputFormat for job, job initialization fails with following exception :
2009-04-28 14:10:40,162 ERROR mapred.EagerTaskInitializationListener (EagerTaskInitializationListener.java:run(83)) - Job initialization failed:
java.lang.IllegalArgumentException: Network location name contains /: /default-rack
at org.apache.hadoop.net.NodeBase.set(NodeBase.java:76)
at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2342)
at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2336)
at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:344)
at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:441)
at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:81)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

When I changed CombineFileInputFormat to pass just rackname (without '/'), JT wrongly resolves the node as /default-rack/<rack-name>.

Solution is to pass hostnames holding the block(on the rack), instead of rackname.

Issue Links

• blocks: MAPREDUCE-364 Change org.apache.hadoop.examples.MultiFileWordCount to use new mapreduce api.
• relates to: MAPREDUCE-1112 Fix CombineFileInputFormat for hadoop 0.20