Hama
  1. Hama
  2. HAMA-367

Runtime Compression of BSP Messages to Improve the Performance

    Details

      Description

      As you know, the exchanging data between processes, is a core part of whole performance in Bulk Synchronous Parallel.

      In this research, we investigate BSP message data compression in the context of large-scale distributed message-passing systems to reduce the communication time of individual messages and to improve the bandwidth of the overall system.

      1. test_files.tar.gz
        2.15 MB
        Ashish Agarwal
      2. HAMA-367.patch
        4 kB
        Apurv Verma
      3. HAMA-367.patch
        10 kB
        Apurv Verma
      4. HAMA-367.patch
        12 kB
        Apurv Verma
      5. HAMA-367_2.patch
        36 kB
        Thomas Jungblut

        Activity

        Hide
        Edward J. Yoon added a comment -
        Show
        Edward J. Yoon added a comment - FYI, http://code.google.com/p/snappy/
        Hide
        Edward J. Yoon added a comment -

        I'm mentoring this project.

        • name: Edward J. Yoon
        • link_id: edwardyoon
        Show
        Edward J. Yoon added a comment - I'm mentoring this project. name: Edward J. Yoon link_id: edwardyoon
        Hide
        Ashish Agarwal added a comment -

        Hi,

        I am interested in working on this project as part of the GSoC. I have prior experience in Java and have also implemented a JPEG compression project and completed a research project on stereo image compression.

        Can you please give me some info as to how I can get started on this project ?

        Thanks
        Ashish

        Show
        Ashish Agarwal added a comment - Hi, I am interested in working on this project as part of the GSoC. I have prior experience in Java and have also implemented a JPEG compression project and completed a research project on stereo image compression. Can you please give me some info as to how I can get started on this project ? Thanks Ashish
        Hide
        Edward J. Yoon added a comment - - edited

        Hi,

        Here's very nice paper:

        http://www.civ.cvut.cz/others/konference_supercomputing/Proceedings_SC_2004/schedule/pdfs/pap183.pdf

        The goal of this project also is a "the compression algorithm that allows cMPI to make better use of available network bandwidth, and several performance-enhancing optimizations."

        As mentioned in paper, the use of suitable compression algorithm for message data type can significantly affect the BSP program performance.

        First of all, you will need to design your own scenario. (1. Why) If you want to improve the performance of exist examples (e.g., Pi Estimator), it's OK. and then, implement the compression. (2. What/How) Finally, proof your work. (3. Benchmark and Conclusion)

        Your code can be improved later. Don't waste your time to writing beautiful code. *The introduction, development, turn and conclusion are important.

        Show
        Edward J. Yoon added a comment - - edited Hi, Here's very nice paper: http://www.civ.cvut.cz/others/konference_supercomputing/Proceedings_SC_2004/schedule/pdfs/pap183.pdf The goal of this project also is a "the compression algorithm that allows cMPI to make better use of available network bandwidth, and several performance-enhancing optimizations." As mentioned in paper, the use of suitable compression algorithm for message data type can significantly affect the BSP program performance. First of all, you will need to design your own scenario. (1. Why) If you want to improve the performance of exist examples (e.g., Pi Estimator), it's OK. and then, implement the compression. (2. What/How) Finally, proof your work. (3. Benchmark and Conclusion) Your code can be improved later. Don't waste your time to writing beautiful code. *The introduction, development, turn and conclusion are important.
        Hide
        Ashish Agarwal added a comment -

        Hi,

        I did some research

        • looked at the paper above
        • read chapter 1 of the book Parallel Scientific Computing - A structured approach ... by R.H.Bisseling
        • looked at the paper "Adaptive-Compi: Enhancing Mpi-Based Applications’ Performance and Scalability by using Adaptive Compression"

        and some online tutorials about MPI.

        In the project, we would be looking at two areas -

        1. Study different compression algorithms and implement the ones that are able to exploit the unique characteristics of MPI messages.

        2. Develop an adaptive compression algorithm - where we could first identify the datatype of the file that is to be compressed and then use the appropriate compression.

        Thanks
        Ashish

        Show
        Ashish Agarwal added a comment - Hi, I did some research looked at the paper above read chapter 1 of the book Parallel Scientific Computing - A structured approach ... by R.H.Bisseling looked at the paper "Adaptive-Compi: Enhancing Mpi-Based Applications’ Performance and Scalability by using Adaptive Compression" and some online tutorials about MPI. In the project, we would be looking at two areas - 1. Study different compression algorithms and implement the ones that are able to exploit the unique characteristics of MPI messages. 2. Develop an adaptive compression algorithm - where we could first identify the datatype of the file that is to be compressed and then use the appropriate compression. Thanks Ashish
        Hide
        Edward J. Yoon added a comment -

        +1

        Show
        Edward J. Yoon added a comment - +1
        Hide
        Thomas Jungblut added a comment -

        FYI if you are going for snappy you can have a look at the Hadoop codec that uses a JNI layer.
        http://code.google.com/p/hadoop-snappy/

        Show
        Thomas Jungblut added a comment - FYI if you are going for snappy you can have a look at the Hadoop codec that uses a JNI layer. http://code.google.com/p/hadoop-snappy/
        Show
        Edward J. Yoon added a comment - See also, https://issues.apache.org/jira/browse/HAMA-380
        Hide
        Ashish Agarwal added a comment -

        I'm assuming we prefer a higher speed of compression than a higher compression ratio. Do we transfer only text and images or do we transfer video and audio as well ?

        I was looking at the LZO compression library. Would that be a good idea ?

        Show
        Ashish Agarwal added a comment - I'm assuming we prefer a higher speed of compression than a higher compression ratio. Do we transfer only text and images or do we transfer video and audio as well ? I was looking at the LZO compression library. Would that be a good idea ?
        Hide
        Thomas Jungblut added a comment -

        Hehe it's a pretty good idea to transfer videos in several supersteps

        I believe we are just transfering byte code. I believe there was a licensing problem with LZO, Hadoop once had to remove this.

        Show
        Thomas Jungblut added a comment - Hehe it's a pretty good idea to transfer videos in several supersteps I believe we are just transfering byte code. I believe there was a licensing problem with LZO, Hadoop once had to remove this.
        Hide
        Edward J. Yoon added a comment -

        To ashish,

        >> I'm assuming we prefer a higher speed of compression than a higher compression ratio.

        Well ... I think, the "high compression ratio" is more important than the speed of compression. The message compression is a part of local computation and will be used to saves network traffic and enhances speed of communication.

        >> Do we transfer only text and images or do we transfer video and audio as well ?

        First of all, the data size is more important factor than the data type.

        Compress only large messages, skip many small messages! It would be nice if we can specify the minimum size of the messages for which compression must be applied. For example,

          <property>
            <name>min.compression.size</name>
            <value>10</value>
            <description>minimum size in KB .... 
            </description>
          </property>
        
        Show
        Edward J. Yoon added a comment - To ashish, >> I'm assuming we prefer a higher speed of compression than a higher compression ratio. Well ... I think, the "high compression ratio" is more important than the speed of compression. The message compression is a part of local computation and will be used to saves network traffic and enhances speed of communication. >> Do we transfer only text and images or do we transfer video and audio as well ? First of all, the data size is more important factor than the data type. Compress only large messages, skip many small messages! It would be nice if we can specify the minimum size of the messages for which compression must be applied. For example, <property> <name>min.compression.size</name> <value>10</value> <description>minimum size in KB .... </description> </property>
        Hide
        Thomas Jungblut added a comment -

        What about tar'ing small files and then compressing?

        Show
        Thomas Jungblut added a comment - What about tar'ing small files and then compressing?
        Hide
        Ashish Agarwal added a comment -

        I am comparing the performance of different compression algorithms right now. Do we have some sample data we would like to use ?

        Show
        Ashish Agarwal added a comment - I am comparing the performance of different compression algorithms right now. Do we have some sample data we would like to use ?
        Hide
        Edward J. Yoon added a comment -

        Hi,

        BTW, have you checked HAMA-380 issue out?

        >> Do we have some sample data we would like to use ?

        http://wiki.apache.org/hama/SSSP

        I think, you can use this sample data.

        Show
        Edward J. Yoon added a comment - Hi, BTW, have you checked HAMA-380 issue out? >> Do we have some sample data we would like to use ? http://wiki.apache.org/hama/SSSP I think, you can use this sample data.
        Hide
        Ashish Agarwal added a comment -

        yes, I am studying Hama-380.

        Show
        Ashish Agarwal added a comment - yes, I am studying Hama-380.
        Hide
        Edward J. Yoon added a comment -

        How's going your work?

        You should have to look at "BSPPeer" class.

        And, I found simple example - http://edwin.baculsoft.com/2010/10/how-to-compress-a-java-object/

        Show
        Edward J. Yoon added a comment - How's going your work? You should have to look at "BSPPeer" class. And, I found simple example - http://edwin.baculsoft.com/2010/10/how-to-compress-a-java-object/
        Hide
        Ashish Agarwal added a comment -

        Hi, I have written some code related to compression and tested against 19 different files of different types and sizes.

        The best case scenario is a compression rate of 90%. Results (before compression, after compression) are -
        alice29.txt(148.5, 51)
        bib (108.7, 33.9)
        book1 (750, 299)
        book2 (596.5, 197.6)
        geo (100, 57.2)
        news (368.3, 137.9)
        obj1 (21, 8.8)
        obj2 (241, 69.3)
        paper1 (51.9, 17.8)
        paper2 (80.3, 28.5)
        paper3 (45.4, 17.3)
        paper4 (13, 5.2)
        paper5 (11.7, 4.7)
        paper6 (37.2, 12.6)
        pic (501.2, 42.8)
        progc (38.7, 12.8)
        progl (70, 15.6)
        progp (48.2, 10.7)
        trans (91.5, 17.9)

        I am asking questions about HAMA-380 on the mailing list.

        Thanks
        Ashish

        Show
        Ashish Agarwal added a comment - Hi, I have written some code related to compression and tested against 19 different files of different types and sizes. The best case scenario is a compression rate of 90%. Results (before compression, after compression) are - alice29.txt(148.5, 51) bib (108.7, 33.9) book1 (750, 299) book2 (596.5, 197.6) geo (100, 57.2) news (368.3, 137.9) obj1 (21, 8.8) obj2 (241, 69.3) paper1 (51.9, 17.8) paper2 (80.3, 28.5) paper3 (45.4, 17.3) paper4 (13, 5.2) paper5 (11.7, 4.7) paper6 (37.2, 12.6) pic (501.2, 42.8) progc (38.7, 12.8) progl (70, 15.6) progp (48.2, 10.7) trans (91.5, 17.9) I am asking questions about HAMA-380 on the mailing list. Thanks Ashish
        Hide
        Ashish Agarwal added a comment -

        The attached file just contains the original and the compressed files.

        Show
        Ashish Agarwal added a comment - The attached file just contains the original and the compressed files.
        Hide
        Edward J. Yoon added a comment -

        I'll look at this tommorow!

        Show
        Edward J. Yoon added a comment - I'll look at this tommorow!
        Hide
        Ashish Agarwal added a comment -

        I am working on the zlib and bzip2 compression right now. I'll have the comparison in a couple of days and I'll post it then.

        Show
        Ashish Agarwal added a comment - I am working on the zlib and bzip2 compression right now. I'll have the comparison in a couple of days and I'll post it then.
        Hide
        Edward J. Yoon added a comment -

        Could you please share about your plan?

        Show
        Edward J. Yoon added a comment - Could you please share about your plan?
        Hide
        Ashish Agarwal added a comment -

        There is a Apache Ant implementation in java of Bzip2 algorithm at - http://www.kohsuke.org/bzip2//
        Zlib compression can be implemented in java using java.util.zip.Deflater

        I plan to compress different files and then compare the compressed file sizes using both algorithms.

        Show
        Ashish Agarwal added a comment - There is a Apache Ant implementation in java of Bzip2 algorithm at - http://www.kohsuke.org/bzip2// Zlib compression can be implemented in java using java.util.zip.Deflater I plan to compress different files and then compare the compressed file sizes using both algorithms.
        Hide
        Edward J. Yoon added a comment -

        Please see the below code block.

            while (it.hasNext()) {
              Entry<InetSocketAddress, ConcurrentLinkedQueue<BSPMessage>> entry = it
                  .next();
        
              BSPPeerInterface peer = peers.get(entry.getKey());
              if (peer == null) {
                peer = getBSPPeerConnection(entry.getKey());
              }
              Iterable<BSPMessage> messages = entry.getValue();
              BSPMessageBundle bundle = new BSPMessageBundle();
              for (BSPMessage message : messages) {
                bundle.addMessage(message);
              }
              peer.put(bundle);
            }
        

        Here's my suggestion for you.

        1. write messages to file somewhere in local disk.
        2. Create compress/decompress methods
        3. Send

        Show
        Edward J. Yoon added a comment - Please see the below code block. while (it.hasNext()) { Entry<InetSocketAddress, ConcurrentLinkedQueue<BSPMessage>> entry = it .next(); BSPPeerInterface peer = peers.get(entry.getKey()); if (peer == null ) { peer = getBSPPeerConnection(entry.getKey()); } Iterable<BSPMessage> messages = entry.getValue(); BSPMessageBundle bundle = new BSPMessageBundle(); for (BSPMessage message : messages) { bundle.addMessage(message); } peer.put(bundle); } Here's my suggestion for you. 1. write messages to file somewhere in local disk. 2. Create compress/decompress methods 3. Send
        Hide
        Edward J. Yoon added a comment -

        (Sorry, my enter key pressed.)

        Anyway, you have to change above code part.

        Show
        Edward J. Yoon added a comment - (Sorry, my enter key pressed.) Anyway, you have to change above code part.
        Hide
        Thomas Jungblut added a comment -

        Sorry for my annoyance (it's not my project), but what do you expect when compressing something with bzip2 and zip?
        I am 100% sure that bzip2 has the higher compression rate yielding into smaller files.

        The best thing overall would be that the compression codec can be configured. So you have to create an interface that compresses a chunk of data, and then provide implementations- for example of BZIP2 (high compression, but longer runtime) or LZO(low compression, but shorter runtime).
        And then this would be applied to the piece of code Edward posted.

        Show
        Thomas Jungblut added a comment - Sorry for my annoyance (it's not my project), but what do you expect when compressing something with bzip2 and zip? I am 100% sure that bzip2 has the higher compression rate yielding into smaller files. The best thing overall would be that the compression codec can be configured. So you have to create an interface that compresses a chunk of data, and then provide implementations- for example of BZIP2 (high compression, but longer runtime) or LZO(low compression, but shorter runtime). And then this would be applied to the piece of code Edward posted.
        Hide
        Ashish Agarwal added a comment -

        I added a couple of functions to BSPPeer.java - one to compress and one to decompress using the bzip2 compression.

        The compressed file will be created on the local disk.

        I'm calling the function just before adding the message to bundle

        for (BSPMessage message : messages) {
        compress(message, "compressedFile.txt");
        bundle.addMessage(message);

        Then I added a breakpoint at the above line and then tried running PIEstimator. But somehow, the program does not stop at the point. Why does it not stop ?

        Also when we add the message in the addMessage function, do I change it so that the input parameter is String compressedFilename and not BSPMessage message ?

        Show
        Ashish Agarwal added a comment - I added a couple of functions to BSPPeer.java - one to compress and one to decompress using the bzip2 compression. The compressed file will be created on the local disk. I'm calling the function just before adding the message to bundle for (BSPMessage message : messages) { compress(message, "compressedFile.txt"); bundle.addMessage(message); Then I added a breakpoint at the above line and then tried running PIEstimator. But somehow, the program does not stop at the point. Why does it not stop ? Also when we add the message in the addMessage function, do I change it so that the input parameter is String compressedFilename and not BSPMessage message ?
        Hide
        Thomas Jungblut added a comment -

        Hey Ashish, as already mentioned on the mailing list. If you are in local mode, you'll never reach this piece of code.
        I can give you an advice to use the MiniBSPCluster in the test package. You have to pull it out of the JUnit test logic, but after setting it up you can submit your jobs quite easily and debug.

        Have a try. If you need help, don't hesitate to write to the mailing list.

        Show
        Thomas Jungblut added a comment - Hey Ashish, as already mentioned on the mailing list. If you are in local mode, you'll never reach this piece of code. I can give you an advice to use the MiniBSPCluster in the test package. You have to pull it out of the JUnit test logic, but after setting it up you can submit your jobs quite easily and debug. Have a try. If you need help, don't hesitate to write to the mailing list.
        Hide
        Edward J. Yoon added a comment -

        Let's break this down into smaller tasks.

        Basically, the large messages should be stored on a local disk before considering compression. Otherwise, we won't free from limited memory.

        Show
        Edward J. Yoon added a comment - Let's break this down into smaller tasks. Basically, the large messages should be stored on a local disk before considering compression. Otherwise, we won't free from limited memory.
        Hide
        Edward J. Yoon added a comment -

        Let's move this to 0.4.0.

        Show
        Edward J. Yoon added a comment - Let's move this to 0.4.0.
        Hide
        Apurv Verma added a comment -

        Please find the patch for HAMA-367 attached, please suggest corrections and improvements.

        Show
        Apurv Verma added a comment - Please find the patch for HAMA-367 attached, please suggest corrections and improvements.
        Hide
        Apurv Verma added a comment -

        Please find the patch for HAMA-367 in the attachments. Please suggest corrections and improvements.

        Show
        Apurv Verma added a comment - Please find the patch for HAMA-367 in the attachments. Please suggest corrections and improvements.
        Hide
        Thomas Jungblut added a comment -

        Looks good to me, was it that easy? However it seems that you haven't included the "BSPCompressedBundle".

        You can make "svn stat" on the project root and see what you haven't included yet, it has a question mark on the left.
        Then you can call "svn add <the file or dir>" and diff the patch again.

        It would be nice if you can include this before I start to test this.

        How expensive is "BSPMessageCompressor.getCompressionRatio()"?
        Does it just compare the byte[] length?

        Show
        Thomas Jungblut added a comment - Looks good to me, was it that easy? However it seems that you haven't included the "BSPCompressedBundle". You can make "svn stat" on the project root and see what you haven't included yet, it has a question mark on the left. Then you can call "svn add <the file or dir>" and diff the patch again. It would be nice if you can include this before I start to test this. How expensive is "BSPMessageCompressor.getCompressionRatio()"? Does it just compare the byte[] length?
        Hide
        Apurv Verma added a comment -

        The new modified patch for 367.

        Show
        Apurv Verma added a comment - The new modified patch for 367.
        Hide
        Thomas Jungblut added a comment -

        Good work! With snappy it seems to work for me.
        However, we would like to have multiple compressors and a user defines this via configuration.
        So I guess we need a compressor/decompressor interface and the typical reflection based class instantiation. But I think we just make a follow-up issue for this.

        Another thing would be testcases, you should include one

        Did you use the code formatter (http://incubator.apache.org/hama/hama-eclipse-formatter.xml)? I think there might be tabs instead of 3 spaces.

        Show
        Thomas Jungblut added a comment - Good work! With snappy it seems to work for me. However, we would like to have multiple compressors and a user defines this via configuration. So I guess we need a compressor/decompressor interface and the typical reflection based class instantiation. But I think we just make a follow-up issue for this. Another thing would be testcases, you should include one Did you use the code formatter ( http://incubator.apache.org/hama/hama-eclipse-formatter.xml)? I think there might be tabs instead of 3 spaces.
        Hide
        Edward J. Yoon added a comment -

        Great! Let's schedule this to 0.5 roadmap.

        Show
        Edward J. Yoon added a comment - Great! Let's schedule this to 0.5 roadmap.
        Hide
        Apurv Verma added a comment -

        1) Used Hama-Eclipse code formatter
        2) Included unit test
        3) Currently uses just snappy as the compressor/decompressor.

        Show
        Apurv Verma added a comment - 1) Used Hama-Eclipse code formatter 2) Included unit test 3) Currently uses just snappy as the compressor/decompressor.
        Hide
        Apurv Verma added a comment -

        Please suggest what would be better as the next step
        1) A compressor decompressor interface like in hadoop or
        2) A simple factory class that does a reflection based instantiation of the compression input/output stream classes which would be given in the configuration.

        So the user would need to put these properties in the conf files.

        hama.bsp.message.compressioninputstream & hama.bsp.message.compressionoutputstream

        The values of these properties would be the fully classified name for the compression input/output stream classes.

        Show
        Apurv Verma added a comment - Please suggest what would be better as the next step 1) A compressor decompressor interface like in hadoop or 2) A simple factory class that does a reflection based instantiation of the compression input/output stream classes which would be given in the configuration. So the user would need to put these properties in the conf files. hama.bsp.message.compressioninputstream & hama.bsp.message.compressionoutputstream The values of these properties would be the fully classified name for the compression input/output stream classes.
        Hide
        Edward J. Yoon added a comment -

        I'm +1 to configurable design.

        Should we add some setter/getter to set/get MessageCompressorClass to BSPJob? If so, we should add setCompressMessage(boolean); too.

        Show
        Edward J. Yoon added a comment - I'm +1 to configurable design. Should we add some setter/getter to set/get MessageCompressorClass to BSPJob? If so, we should add setCompressMessage(boolean); too.
        Hide
        Thomas Jungblut added a comment -

        Hi Apurv, sorry I lost track of the whole issue.

        But I had a bit more time to code it to a configurable design.

        Are you okay with this? Or do you have any suggestions?
        And was there a specific reason you picked the 0.3.0 snapshot of snappy instead of the 0.2.0 release?

        And I added BZIP2 Codec from Hadoop.

        Show
        Thomas Jungblut added a comment - Hi Apurv, sorry I lost track of the whole issue. But I had a bit more time to code it to a configurable design. Are you okay with this? Or do you have any suggestions? And was there a specific reason you picked the 0.3.0 snapshot of snappy instead of the 0.2.0 release? And I added BZIP2 Codec from Hadoop.
        Hide
        Apurv Verma added a comment -

        Hi Thomas,
        Apologies from my side also for not completing it completely.

        I saw the changes. Looks perfect to me, everything seems to be taken care of. We no longer need

        setCompressMessage(boolean)
        

        in BSPJob as Edward had asked. The user can choose not to specify this parameter "hama.messenger.compression.class" in site.xml in which case

         CompressableMessageManager 

        would return a null and no compressor would be created.

        No, snappy 0.2.0 should have been used, my mistake.

        thanks, your design taught me many things.

        +1 (It has been a long issue, I hope this is resolved now)

        Show
        Apurv Verma added a comment - Hi Thomas, Apologies from my side also for not completing it completely. I saw the changes. Looks perfect to me, everything seems to be taken care of. We no longer need setCompressMessage(boolean) in BSPJob as Edward had asked. The user can choose not to specify this parameter "hama.messenger.compression.class" in site.xml in which case CompressableMessageManager would return a null and no compressor would be created. No, snappy 0.2.0 should have been used, my mistake. thanks, your design taught me many things. +1 (It has been a long issue, I hope this is resolved now)
        Hide
        Thomas Jungblut added a comment -

        Thanks for your contribution Apurv, you saved us a lot of time and nerves.

        Just committed it.

        would return a null and no compressor would be created.

        In any way (but the case where the default compressor is not found in classpath) it returns the default compressor. In our case this is snappy.
        It has a good tradeoff between amount of compression and time to compress. So there should be no problem.

        Great we can now close this! -> Fixed!

        Show
        Thomas Jungblut added a comment - Thanks for your contribution Apurv, you saved us a lot of time and nerves. Just committed it. would return a null and no compressor would be created. In any way (but the case where the default compressor is not found in classpath) it returns the default compressor. In our case this is snappy. It has a good tradeoff between amount of compression and time to compress. So there should be no problem. Great we can now close this! -> Fixed!
        Hide
        Hudson added a comment -

        Integrated in Hama-Nightly #462 (See https://builds.apache.org/job/Hama-Nightly/462/)
        HAMA-367: Runtime Compression of BSP Messages to Improve the Performance (Revision 1291977)

        Result = SUCCESS
        tjungblut :
        Files :

        • /incubator/hama/trunk/CHANGES.txt
        • /incubator/hama/trunk/core/pom.xml
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AvroBSPMessageBundle.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AvroMessageManagerImpl.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/CompressableMessageManager.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManager.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManagerImpl.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManagerFactory.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/BSPCompressedBundle.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/BSPMessageCompressor.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/BSPMessageCompressorFactory.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/Bzip2Compressor.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/SnappyCompressor.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/util/CompressionUtil.java
        • /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/compress
        • /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/compress/TestBSPCompressBundle.java
        • /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/compress/TestBSPMessageCompressor.java
        • /incubator/hama/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in Hama-Nightly #462 (See https://builds.apache.org/job/Hama-Nightly/462/ ) HAMA-367 : Runtime Compression of BSP Messages to Improve the Performance (Revision 1291977) Result = SUCCESS tjungblut : Files : /incubator/hama/trunk/CHANGES.txt /incubator/hama/trunk/core/pom.xml /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AvroBSPMessageBundle.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AvroMessageManagerImpl.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/CompressableMessageManager.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManager.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManagerImpl.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManagerFactory.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/BSPCompressedBundle.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/BSPMessageCompressor.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/BSPMessageCompressorFactory.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/Bzip2Compressor.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/compress/SnappyCompressor.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/util/CompressionUtil.java /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/compress /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/compress/TestBSPCompressBundle.java /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/compress/TestBSPMessageCompressor.java /incubator/hama/trunk/pom.xml

          People

          • Assignee:
            Apurv Verma
            Reporter:
            Edward J. Yoon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2,016h
              2,016h
              Remaining:
              Remaining Estimate - 2,016h
              2,016h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development