[PIG-2459] Can't run penny in MapReduce mode

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.9.1
• Labels: newbie

Environment

Testing in a cluster with 8 node
Ubuntu 11.10 64bit
java 1.6
Hadoop 0.20.203

• Assignee: Unassigned
• Reporter: Andy He
• Created: 06/Jan/12 07:02
• Updated: 28/Nov/13 22:45
• Resolved: 28/Nov/13 22:45

Description

Hi folks,

I have problems to run penny in my clusters.

First of all, the hadoop cluster is fine, and the pig program performs well.

Yet, when I'm trying to run the penny tool in the MapReduce Mode with the command:
java -cp pig-0.9.1/contrib/penny/java/penny.jar:pig-0.9.1/pig-0.9.1.jar:$HADOOP_CONF_DIR org.apache.pig.penny.apps.ds.Main test.pig

I get the following errors:

===========
INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://lotr4.comp.polyu.edu.hk:9000
Exception in thread "main" java.lang.RuntimeException: Failed to create DataStorage
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:203)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:118)
at org.apache.pig.impl.PigContext.connect(PigContext.java:185)
at org.apache.pig.PigServer.<init>(PigServer.java:244)
at org.apache.pig.PigServer.<init>(PigServer.java:229)
at org.apache.pig.tools.ToolsPigServer.<init>(ToolsPigServer.java:70)
at org.apache.pig.penny.ParsedPigScript.<init>(ParsedPigScript.java:82)
at org.apache.pig.penny.PennyServer.parse(PennyServer.java:44)
at org.apache.pig.penny.apps.ds.Main.main(Main.java:36)
Caused by: java.io.IOException: Call to lotr4.comp.polyu.edu.hk/158.132.10.162:9000 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 10 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
===========

The test.pig script just loads a file and then stores it back:
-----------
data = LOAD 'input/student';
STORE data INTO 'output';
-----------

Is it the problem that I miss some environmental variables for penny? Hope someone can help.

Thanks