[PIG-2198] pig.additional.jars does not support s3 paths

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.8.1
• Release Note: Changed the delimiter from : to
• Assignee: Unassigned
• Reporter: Jeremy Custenborder
• Created: 01/Aug/11 15:26
• Updated: 17/Aug/11 19:07

Description

The delimiter for additional jars is ":" which is part of the uri for a S3 path.

Repro
Add the following to the command line.

-Dpig.additional.jars=s3://bucket/udf.jar

Error before Pig is launched
----------------------------
ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3

org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Failed to register jar :s3. Caught exception.
at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:253)
at org.apache.pig.PigServer.<init>(PigServer.java:239)
at org.apache.pig.PigServer.<init>(PigServer.java:215)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
at org.apache.pig.Main.run(Main.java:492)
at org.apache.pig.Main.main(Main.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 101: Could not copy s3 to local destination /mnt/var/lib/hadoop/steps/1/tmp/s3
at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:765)
at org.apache.pig.PigServer.registerJar(PigServer.java:483)
at org.apache.pig.PigServer.addJarsFromProperties(PigServer.java:248)
... 10 more
Caused by: java.io.FileNotFoundException: File does not exist: s3
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1256)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1237)
at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:763)
... 12 more
================================================================================