[HADOOP-6178] GenericOptionsParser should ignore symlinks from archive filenames

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: util
• Assignee: Unassigned
• Reporter: Amar Kamat
• Created: 07/Aug/09 09:47
• Updated: 22/Feb/10 03:46
• Resolved: 22/Feb/10 03:46

Description

http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#Large+files+and+archives+in+Hadoop+Streaming example doesnt work as expected.
Stack trace :

trunk$  ./bin/hadoop jar build/contrib/streaming/hadoop-0.21.0-dev-streaming.jar  -D mapred.map.tasks=1 \
         -D mapred.reduce.tasks=1 -D mapred.job.name="Experiment" \
        -archives 'hdfs://namenode:port/user/me/cachedir.jar#testlink' -input input -output output -mapper "xargs cat" -reducer "cat"
java.io.FileNotFoundException: File hdfs://namenode:port/user/me/cachedir.jar#testlink does not exist.
	at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:349)
	at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)
	at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:375)
	at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
	at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:138)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

Issue Links

• relates to: MAPREDUCE-767 to remove mapreduce dependency on commons-cli2