Uploaded image for project: 'Hama'
  1. Hama
  2. HAMA-992

Hama streaming

    XMLWordPrintableJSON

Details

    • Question
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.7.1
    • None
    • bsp core, pipes

    • RASPBIAN JESSIE
      Full desktop image based on Debian Jessie

    • Important

    Description

      Hello all,

      I am trying to implement apache hama on Raspberry pi model 3 to establish a distributed computing platform for scientific computation. I am trying to run hama streaming over hadoop on a single namenode but I am facing a bit of a difficulty in streaming my python code. I have downloaded the hama streaming repository from :-

      https://github.com/thomasjungblut/HamaStreaming

      I ran the examples and also HelloWorldBSP.py on Hama and they work well. But as soon as I switch to running my python code, the job fails.

      I am trying to run the code with the following command:-

      hama pipes -streaming true -bspTasks 1 -interpreter python -output /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs python.py

      Below is the log file for your reference. I hope you can find time to help me in this minor project:-

      16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001
      16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting
      16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting
      16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting
      16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting
      16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting
      16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting
      16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting
      16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer address:localhost port:61001
      16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files
      16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client
      16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At localhost/127.0.0.1:61001
      java.lang.NumberFormatException: For input string: "Traceback (most recent call last):"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Integer.parseInt(Integer.java:580)
      at java.lang.Integer.parseInt(Integer.java:615)
      at org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174)
      at org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106)
      16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad command code: -2
      at org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174)

      java.util.concurrent.BrokenBarrierException
      at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
      at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
      at org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223)
      at org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293)
      at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43)
      at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
      at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
      at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
      Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: java.lang.Exception: Bad command code: -2
      at org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182)
      Caused by: java.lang.Exception: Bad command code: -2
      at org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174)
      16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
      java.io.IOException: Stream closed
      at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
      at java.io.OutputStream.write(OutputStream.java:116)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
      at java.io.DataOutputStream.flush(DataOutputStream.java:123)
      at org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
      at org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
      at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
      at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
      at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
      at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
      16/06/01 14:48:52 ERROR bsp.BSPTask: Error cleaning up after bsp executed.
      java.io.IOException: Stream closed
      at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
      at java.io.OutputStream.write(OutputStream.java:116)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
      at java.io.DataOutputStream.flush(DataOutputStream.java:123)
      at org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
      at org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271)
      at org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95)
      at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
      at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
      at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
      16/06/01 14:48:53 INFO ipc.Server: Stopping server on 61001
      16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 1 on 61001: exiting
      16/06/01 14:48:53 INFO ipc.Server: Stopping IPC Server listener on 61001
      16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 0 on 61001: exiting
      16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 3 on 61001: exiting
      16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 4 on 61001: exiting
      16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 2 on 61001: exiting
      16/06/01 14:48:53 INFO ipc.Server: Stopping IPC Server Responder
      16/06/01 14:48:53 ERROR bsp.BSPTask: Shutting down ping service.
      16/06/01 14:48:53 FATAL bsp.GroomServer: Error running child
      java.io.IOException: Stream closed
      at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
      at java.io.OutputStream.write(OutputStream.java:116)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
      at java.io.DataOutputStream.flush(DataOutputStream.java:123)
      at org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
      at org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
      at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
      at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
      at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
      at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
      java.io.IOException: Stream closed
      at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
      at java.io.OutputStream.write(OutputStream.java:116)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
      at java.io.DataOutputStream.flush(DataOutputStream.java:123)
      at org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
      at org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
      at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
      at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
      at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
      at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #16

          Activity

            People

              Unassigned Unassigned
              chaitanya2692 Chaitanya
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: