[PIG-2745] Pig e2e test RubyUDFs fails in MR mode when running from tarball

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1
• Fix Version/s: 0.11, 0.10.1
• Hadoop Flags: Reviewed
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 09/Jun/12 19:19
• Updated: 06/Jan/13 23:57
• Resolved: 18/Jun/12 01:54

Description

To reproduce the issue, please run the e2e test "RubyUDFs_1" in MR mode from the tarball (not from installed Pig - please see why below). Either pseudo-distributed-mode or full-mode Hadoop can be used.

ant -Dhadoopversion=23 -Dharness.old.pig=`pwd` -Dharness.cluster.conf=/etc/hadoop/conf/ -Dharness.cluster.bin=/usr/lib/hadoop/bin/hadoop test-e2e -Dtests.to.run="-t RubyUDFs_1"

The test fails with the following error:

java.lang.IllegalStateException: Could not initialize interpreter (from file system or classpath) with /home/cheolsoo/pig-0.10/test/e2e/pig/testdist/libexec/ruby/scriptingudfs.rb

Looking at the job jar generated by Pig, "scriptingudfs.rb" can be found as follows:

[cheolsoo@c1405 pig-cheolsoo]$ jar tvf bad.jar | grep scriptingudfs.rb
  2491 Fri Jun 08 15:52:08 PDT 2012 /home/cheolsoo/pig-0.10/test/e2e/pig/testdist/libexec/ruby/scriptingudfs.rb

Looking at getScriptAsStream() method in ScriptEngine.java, "scriptingudfs.rb" is supposed to be read from the job jar, but it is not. The reason is because getResourceAsStream("/x") looks for "x" (without the leading "/") not "/x". Since "scriptingudfs.rb" is stored with it absolute path, it ends up being not found by getResourceAsStream(scriptPath).

File file = new File(scriptPath);
if (file.exists()) {
    try {
        is = new FileInputStream(file);
    } catch (FileNotFoundException e) {
        throw new IllegalStateException("could not find existing file "+scriptPath, e);
    }
} else {
    if (file.isAbsolute()) {
        is = ScriptEngine.class.getResourceAsStream(scriptPath);
    } else {
        is = ScriptEngine.class.getResourceAsStream("/" + scriptPath);
    }
}

In fact, the test passes if you run in local mode or from installed Pig. The reason is because "scriptingudfs.rb" is found in local file system (e.g /usr/share/pig/test/e2e/pig/udfs/ruby/scriptingudfs.rb).

The fix seems straightforward. Attached is the patch that removes the leading "/" when registering UDF scripts so that they are stored without the leading "/" in the job jar as follows:

[cheolsoo@c1405 pig-cheolsoo]$ jar tvf good.jar | grep scriptingudfs.rb
  2491 Fri Jun 08 15:52:08 PDT 2012 home/cheolsoo/pig-0.10/test/e2e/pig/testdist/libexec/ruby/scriptingudfs.rb

Thanks!

Issue Links

• relates to: PIG-2760 resources added with a relative path are added to the JobXXXX jar file under their absolute path
• relates to: PIG-2761 With hadoop23 importing modules inside python script does not work
• relates to: PIG-2623 Support S3 paths for registering scripting UDFs.