Hi to all,
this is my first mail to this mailing list so please have patience if I
make some bad choice in the format.
I have a kubuntu-14.04 on an old Intel Core 2 Duo processor T7500 with 2 GB
of RAM.
I have properly installed Hadoop-2.7.1, Sun Java JDK 1.8.0_60 e Hama-0.7.0
as you can see from the following lines:
>
> > $ hadoop version
> > Hadoop 2.7.1
> > Subversion https:15ecc87ccf4a0228f35af08fc56de536e6ce657a
> > Compiled by jenkins on 2015-06-29T06:04Z
> > Compiled with protoc 2.5.0
> > From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
> > This command was run using
/home/tora/Downloads/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar
> > $ java -version
> > java version "1.8.0_60"
> > Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> >
I am able to properly run the basic Hadoop and Hama examples like the
following:
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar
grep input output 'dfs[a-z.]+'
bin/hama jar hama-examples-0.7.0.jar pi
My problem is that I receive the following error message when I try to run
the HelloWorld Hama Streaming example with this instruction:
bin/hama pipes -streaming true -bspTasks 2 -interpreter python3.2
-cachefiles /tmp/PyStreaming/*.py -output /tmp/pystream-out/ -program
/tmp/PyStreaming/BSPRunner.py -programArgs HelloWorldBSP
The default python interpreters of my OS were python-2.7 and python-3.4;
since I had problems with this example I also tried to install python-3.2
with the following instructions but it didn't solved the problem:
>
> > sudo apt-get install software-properties-common
> > sudo apt-add-repository ppa:fkrull/deadsnakes
> > sudo apt-get update
> > sudo apt-get install python3.2
> >
The installed python version is 3.2.6 as you can see from the following
lines:
>
> > $ python3.2
> > Python 3.2.6 (default, Oct 21 2014, 12:50:03)
> > [GCC 4.8.2] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>>
> >
The error message is the following (I am working in local mode so I didn't
run bin/start-bspd.sh):
>
>
> >
> > $ clear;bin/hama pipes -streaming true -bspTasks 2 -interpreter
python3.2 -cachefiles /tmp/PyStreaming/*.py -output /tmp/pystream-out/
-program /tmp/PyStreaming/BSPRunner.py -programArgs HelloWorldBSP
> > 15/09/30 12:39:11 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
> > 15/09/30 12:39:11 INFO pipes.Submitter: Streaming enabled!
> > 15/09/30 12:39:11 INFO Configuration.deprecation: fs.default.name is
deprecated. Instead, use fs.defaultFS
> > 15/09/30 12:39:11 INFO Configuration.deprecation: user.name is
deprecated. Instead, use mapreduce.job.user.name
> > 15/09/30 12:39:11 WARN conf.Configuration:
org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4ddced80:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
> > 15/09/30 12:39:11 WARN conf.Configuration:
org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4ddced80:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
> > 15/09/30 12:39:12 INFO Configuration.deprecation: user.name is
deprecated. Instead, use mapreduce.job.user.name
> > 15/09/30 12:39:12 INFO bsp.BSPJobClient: Running job:
job_localrunner_0001
> > 15/09/30 12:39:12 INFO Configuration.deprecation:
mapred.cache.localFiles is deprecated. Instead, use
mapreduce.job.cache.local.files
> > 15/09/30 12:39:12 INFO bsp.LocalBSPRunner: Setting up a new barrier for
2 tasks!
> > java.lang.NumberFormatException: For input string: "Traceback (most
recent call last):"
> > at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > at java.lang.Integer.parseInt(Integer.java:580)
> > at java.lang.Integer.parseInt(Integer.java:615)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174)
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106)
> > java.lang.NumberFormatException: For input string: "Traceback (most
recent call last):"
> > at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > at java.lang.Integer.parseInt(Integer.java:580)
> > at java.lang.Integer.parseInt(Integer.java:615)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174)
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106)
> > 15/09/30 12:39:12 ERROR protocol.UplinkReader: java.lang.Exception: Bad
command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> >
> > 15/09/30 12:39:12 ERROR protocol.UplinkReader: java.lang.Exception: Bad
command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> >
> > java.util.concurrent.BrokenBarrierException
> > at
java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
> > at
java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223)
> > at
org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293)
> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException:
java.lang.Exception: Bad command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:180)
> > Caused by: java.lang.Exception: Bad command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> > java.util.concurrent.BrokenBarrierException
> > at
java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
> > at
java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223)
> > at
org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293)
> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException:
java.lang.Exception: Bad command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:180)
> > Caused by: java.lang.Exception: Bad command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Exception during BSP
execution!
> > java.io.IOException: Stream closed
> > at
java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> > at java.io.OutputStream.write(OutputStream.java:116)
> > at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> > at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Exception during BSP
execution!
> > java.lang.InterruptedException: java.lang.Exception: Bad command code:
-2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> >
> > at
org.apache.hama.pipes.protocol.BinaryProtocol.waitForFinish(BinaryProtocol.java:335)
> > at
org.apache.hama.pipes.PipesApplication.waitForFinish(PipesApplication.java:329)
> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:47)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Error cleaning up after bsp
execution.
> > java.io.IOException: Stream closed
> > at
java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> > at java.io.OutputStream.write(OutputStream.java:116)
> > at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> > at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271)
> > at org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Error cleaning up after bsp
execution.
> > java.io.IOException: Stream closed
> > at
java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> > at java.io.OutputStream.write(OutputStream.java:116)
> > at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> > at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> > at
org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271)
> > at org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Exception during BSP
execution!
> > java.util.concurrent.ExecutionException:
java.lang.InterruptedException: java.lang.Exception: Bad command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> >
> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> > at
org.apache.hama.bsp.LocalBSPRunner$ThreadObserver.run(LocalBSPRunner.java:313)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.InterruptedException: java.lang.Exception: Bad
command code: -2
> > at
org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172)
> >
> > at
org.apache.hama.pipes.protocol.BinaryProtocol.waitForFinish(BinaryProtocol.java:335)
> > at
org.apache.hama.pipes.PipesApplication.waitForFinish(PipesApplication.java:329)
> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:47)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
> > at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > ... 1 more
> > 15/09/30 12:39:13 INFO bsp.BSPJobClient: Job failed.
> >
I tried to searching for these error messages with google but I didn't find
any solution.
Does anyone of you have any idea of how can I solve this problem?
Any help is appreciated :)
Thank you
Best
Major