Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.2
    • Component/s: bsp core, messaging, pipes
    • Labels:

      Description

      Adapting Hadoop Pipes to Hama for C++ integration.

      The package org.apache.hama.pipes contains all pipes specific code. The C++ implementation is located in the c++ folder under pipes and utils. There are also some minor modifications in other packages but you can checkout the full Hama source here: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/

      The changelog can be found here: here

      Hama Pipes: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/
      Hama Pipes C++: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/

      Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

      virtual void sendMessage(const string& peerName, const string& msg) = 0;

      Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

      virtual const string& getCurrentMessage() = 0;

      Returns a message from the peer's received messages queue (a FIFO).

      virtual int getNumCurrentMessages() = 0;

      Returns the number of messages in the peer's received messages queue.

      virtual void sync() = 0;

      Barrier Synchronization.
      Sends all the messages in the outgoing message queues to the corresponding remote peers.

      virtual long getSuperstepCount() = 0;

      Returns the count of current super-step

      virtual const string& getPeerName() = 0;

      Returns the name of this peer in the format "hostname:port".

      virtual const string& getPeerName(int index) = 0;

      Returns the name of n-th peer from sorted array by name.

      virtual int getPeerIndex() = 0;

      Returns the index of this peer from sorted array by name.

      virtual vector<string> getAllPeerNames() = 0;

      Returns the names of all the peers executing tasks from the same job (including this peer).

      virtual int getNumPeers() = 0;

      Returns the number of peers

      virtual void clear() = 0;

      Clears all queues entries.

      virtual void write(const string& key, const string& value) = 0;

      Writes a key/value pair to the output collector

      virtual bool readNext(string& key, string& value) = 0;

      Deserializes the next input key value into the given objects;

      virtual void reopenInput() = 0;

      Closes the input and opens it right away, so that the file pointer is at the beginning again.

      1. HAMA-619.patch
        2.78 MB
        Martin Illecker
      2. HAMA-619.patch
        66 kB
        Thomas Jungblut

        Issue Links

          Activity

          Martin Illecker created issue -
          Thomas Jungblut made changes -
          Field Original Value New Value
          Link This issue is related to HAMA-601 [ HAMA-601 ]
          Edward J. Yoon made changes -
          Assignee Martin Illecker [ bafu ]
          Martin Illecker made changes -
          Description Adapting Hadoop Pipes to Hama for C++ integration. *Adapting Hadoop Pipes to Hama for C++ integration.*

          The SVN source is available here: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/
          Hama Pipes: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/
          Hama Pipes C++: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          Martin Illecker made changes -
          Description *Adapting Hadoop Pipes to Hama for C++ integration.*

          The SVN source is available here: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/
          Hama Pipes: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/
          Hama Pipes C++: https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          *Adapting Hadoop Pipes to Hama for C++ integration.*

          Today I finished the first version of Hama Pipes. :-)
          I have added the package *org.apache.hama.pipes* which contains all pipes specific code. The C++ implementation is located in the *c++* folder under *pipes* and *utils*. There are also some minor modifications in other packages but you can checkout the full Hama source here:
          [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]

          The working *revision is 32*, for the changelog see [here|http://hadoop.illecker.at/?page_id=6].

          Hama Pipes: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/]
          Hama Pipes C++: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          Martin Illecker made changes -
          Description *Adapting Hadoop Pipes to Hama for C++ integration.*

          Today I finished the first version of Hama Pipes. :-)
          I have added the package *org.apache.hama.pipes* which contains all pipes specific code. The C++ implementation is located in the *c++* folder under *pipes* and *utils*. There are also some minor modifications in other packages but you can checkout the full Hama source here:
          [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]

          The working *revision is 32*, for the changelog see [here|http://hadoop.illecker.at/?page_id=6].

          Hama Pipes: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/]
          Hama Pipes C++: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          *Adapting Hadoop Pipes to Hama for C++ integration.*

          Today I finished the first version of Hama Pipes. :-)
          I have added the package *org.apache.hama.pipes* which contains all pipes specific code. The C++ implementation is located in the *c++* folder under *pipes* and *utils*. There are also some minor modifications in other packages but you can checkout the full Hama source here: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]

          The working *revision is 32*, for the changelog see [here|http://hadoop.illecker.at/?page_id=6].

          Hama Pipes: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/]
          Hama Pipes C++: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          Martin Illecker made changes -
          Description *Adapting Hadoop Pipes to Hama for C++ integration.*

          Today I finished the first version of Hama Pipes. :-)
          I have added the package *org.apache.hama.pipes* which contains all pipes specific code. The C++ implementation is located in the *c++* folder under *pipes* and *utils*. There are also some minor modifications in other packages but you can checkout the full Hama source here: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]

          The working *revision is 32*, for the changelog see [here|http://hadoop.illecker.at/?page_id=6].

          Hama Pipes: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/]
          Hama Pipes C++: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          *Adapting Hadoop Pipes to Hama for C++ integration.*

          The package *org.apache.hama.pipes* contains all pipes specific code. The C++ implementation is located in the *c++* folder under *pipes* and *utils*. There are also some minor modifications in other packages but you can checkout the full Hama source here: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]

          The changelog can be found here: [here|http://hadoop.illecker.at/?page_id=6]

          Hama Pipes: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/]
          Hama Pipes C++: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]


          Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)

          *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
          bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.

          *{{virtual const string& getCurrentMessage() = 0;}}*
          bq. Returns a message from the peer's received messages queue (a FIFO).

          *{{virtual int getNumCurrentMessages() = 0;}}*
          bq. Returns the number of messages in the peer's received messages queue.

          *{{virtual void sync() = 0;}}*
          {quote}
          Barrier Synchronization.
          Sends all the messages in the outgoing message queues to the corresponding remote peers.
          {quote}

          *{{virtual long getSuperstepCount() = 0;}}*
          bq. Returns the count of current super-step
               
          *{{virtual const string& getPeerName() = 0;}}*
          bq. Returns the name of this peer in the format "hostname:port".
              
          *{{virtual const string& getPeerName(int index) = 0;}}*
          bq. Returns the name of n-th peer from sorted array by name.
              
          *{{virtual int getPeerIndex() = 0;}}*
          bq. Returns the index of this peer from sorted array by name.

          *{{virtual vector<string> getAllPeerNames() = 0;}}*
          bq. Returns the names of all the peers executing tasks from the same job (including this peer).
              
          *{{virtual int getNumPeers() = 0;}}*
          bq. Returns the number of peers
              
          *{{virtual void clear() = 0;}}*
          bq. Clears all queues entries.
              
          *{{virtual void write(const string& key, const string& value) = 0;}}*
          bq. Writes a key/value pair to the output collector
              
          *{{virtual bool readNext(string& key, string& value) = 0;}}*
          bq. Deserializes the next input key value into the given objects;

          *{{virtual void reopenInput() = 0;}}*
          bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
          Thomas Jungblut made changes -
          Attachment HAMA-619.patch [ 12544664 ]
          Edward J. Yoon made changes -
          Fix Version/s 0.7.0 [ 12320349 ]
          Edward J. Yoon made changes -
          Fix Version/s 0.7.0 [ 12320349 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577506 ]
          Edward J. Yoon made changes -
          Fix Version/s 0.7.0 [ 12320349 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577506 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577509 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577509 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577526 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577526 ]
          Martin Illecker made changes -
          Attachment HAMA-619.patch [ 12577534 ]
          Edward J. Yoon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Edward J. Yoon made changes -
          Fix Version/s 0.6.2 [ 12324385 ]
          Fix Version/s 0.7.0 [ 12320349 ]
          Martin Illecker made changes -
          Component/s pipes [ 12321200 ]
          Martin Illecker made changes -
          Labels pipes

            People

            • Assignee:
              Martin Illecker
              Reporter:
              Martin Illecker
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development