[HAMA-619] Hama Pipes

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.6.2
• Component/s: bsp core, messaging, pipes
• Labels: pipes
• Assignee: Martin Illecker
• Reporter: Martin Illecker
• Created: 03/Aug/12 19:22
• Updated: 06/Jan/14 15:24
• Resolved: 09/Apr/13 01:44

Description

Adapting Hadoop Pipes to Hama for C++ integration.

The package org.apache.hama.pipes contains all pipes specific code. The C++ implementation is located in the c++ folder under pipes and utils. There are also some minor modifications in other packages but you can checkout the full Hama source here: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/

The changelog can be found here: here

Hama Pipes: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/
Hama Pipes C++: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/

Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

virtual void sendMessage(const string& peerName, const string& msg) = 0;

Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

virtual const string& getCurrentMessage() = 0;

Returns a message from the peer's received messages queue (a FIFO).

virtual int getNumCurrentMessages() = 0;

Returns the number of messages in the peer's received messages queue.

virtual void sync() = 0;

Barrier Synchronization.
Sends all the messages in the outgoing message queues to the corresponding remote peers.

virtual long getSuperstepCount() = 0;

Returns the count of current super-step

virtual const string& getPeerName() = 0;

Returns the name of this peer in the format "hostname:port".

virtual const string& getPeerName(int index) = 0;

Returns the name of n-th peer from sorted array by name.

virtual int getPeerIndex() = 0;

Returns the index of this peer from sorted array by name.

virtual vector<string> getAllPeerNames() = 0;

Returns the names of all the peers executing tasks from the same job (including this peer).

virtual int getNumPeers() = 0;

Returns the number of peers

virtual void clear() = 0;

Clears all queues entries.

virtual void write(const string& key, const string& value) = 0;

Writes a key/value pair to the output collector

virtual bool readNext(string& key, string& value) = 0;

Deserializes the next input key value into the given objects;

virtual void reopenInput() = 0;

Closes the input and opens it right away, so that the file pointer is at the beginning again.

Issue Links

• is related to: HAMA-601 Hama Streaming