[PIG-2979] Pig.jar doesn't work with hadoop-2.0.x

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11, 0.12.0
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 15/Oct/12 21:56
• Updated: 22/Feb/13 04:54
• Resolved: 09/Nov/12 00:43

Description

To reproduce, please do the following:

1) ensure that no hadoop is installed and therefore no hadoop classes are present in classpath.
2) ant clean jar -Dhadoopversion=23
3) ./bin/pig -x local
4) fail with the following error: (you may not see this if slf4j is available in classpath.)

cheolsoo@localhost:~/workspace/pig-trunk $cat  /home/cheolsoo/workspace/pig-trunk/pig_1350687456711.log
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/slf4j/LoggerFactory

java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
...

5) cp ./build/ivy/lib/Pig/slf4j-api-1.6.1.jar lib
6) ./bin/pig -x local
7) fail with the following error:

cheolsoo@localhost:~/workspace/pig-trunk $cat /home/cheolsoo/workspace/pig-trunk/pig_1350687052995.log
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
	at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
	at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:204)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:117)
	at org.apache.pig.impl.PigContext.connect(PigContext.java:229)
	at org.apache.pig.PigServer.<init>(PigServer.java:213)
	at org.apache.pig.PigServer.<init>(PigServer.java:198)
	at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
	at org.apache.pig.Main.run(Main.java:535)
	at org.apache.pig.Main.main(Main.java:154)
Caused by: java.io.IOException: No FileSystem for scheme: file
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2130)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2137)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2176)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2158)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
	at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
	... 9 more
================================================================================

8) cp ./build/ivy/lib/Pig/hadoop-common-2.0.0-alpha.jar lib
9) ./bin/pig -x local
10) works fine!

In fact, this is also an issue with running e2e test in local mode:

ant clean
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23

The ant test-e2e-local fails with the following error:

java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
        at org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)

Issue Links

• is duplicated by: PIG-2993 Fix local mode on Hadoop-0.23