Hama
  1. Hama
  2. HAMA-524

[GSoC 2012] Sparse Matrix-Vector multiplication (SpMV) on Hama

    Details

      Description

      Implement Efficient and Fast SpMV algorithm which can be widely used in scientific computing, financial modeling, information retrieval, and others, using Hama Bulk Synchronous Parallel framework.

      1. spmv.png
        12 kB
        Edward J. Yoon
      2. SpMV_v01.patch
        45 kB
        Edward J. Yoon
      3. HAMA-524.patch
        49 kB
        Mikalai Parafeniuk
      4. HAMA_524_v3.patch
        61 kB
        Mikalai Parafeniuk
      5. HAMA_524_v2.patch
        62 kB
        Mikalai Parafeniuk
      6. HAMA_524_v1.patch
        61 kB
        Mikalai Parafeniuk
      7. eof_exception.zip
        14 kB
        Mikalai Parafeniuk
      8. broken_pipe_exception.zip
        12 kB
        Mikalai Parafeniuk

        Activity

        Hide
        Thomas Jungblut added a comment -

        I'm going to apply for this over-next week, because I think it is the only really challenging issue of all whole ASF GSoC issues.
        However, I'd be happy if other students will apply for it as well

        Show
        Thomas Jungblut added a comment - I'm going to apply for this over-next week, because I think it is the only really challenging issue of all whole ASF GSoC issues. However, I'd be happy if other students will apply for it as well
        Hide
        Edward J. Yoon added a comment -

        Show
        Edward J. Yoon added a comment -
        Hide
        Alejandro Claudio Esquivel Gonzalez added a comment -

        Hi,

        Show
        Alejandro Claudio Esquivel Gonzalez added a comment - Hi,
        Hide
        Alejandro Claudio Esquivel Gonzalez added a comment -

        Hi, I thought the day to assign students to the projects was until april 23th. I'm already working on this Sparse Matrix-Vector Multiplication project. I'm developing an algorithm to store a Sparse Matrix in the CSC format. I submitted my proposal as you told me but did not get any answer so I continued my research by myself.
        Is this assignation definitive?
        Thanks.

        Show
        Alejandro Claudio Esquivel Gonzalez added a comment - Hi, I thought the day to assign students to the projects was until april 23th. I'm already working on this Sparse Matrix-Vector Multiplication project. I'm developing an algorithm to store a Sparse Matrix in the CSC format. I submitted my proposal as you told me but did not get any answer so I continued my research by myself. Is this assignation definitive? Thanks.
        Hide
        Edward J. Yoon added a comment -

        Alejandro,

        Sorry for my late reply. This was most popular task among entire ASF tasks and I've received many similar proposals. I heard that it's possible to have one student per task from Google guys. So, I had considered breaking this task into pieces but I couldn't find good idea.

        You cannot have the students working as a group:
        thi
        http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2012/faqs#group

        As you know, this is a open source project. If you did already some work on here, Please keep an eye on this task.

        The transient GSoC experience does not represent much to your career. Making a contribution to the project you love or Acquiring ASF committership is more valuable.

        Show
        Edward J. Yoon added a comment - Alejandro, Sorry for my late reply. This was most popular task among entire ASF tasks and I've received many similar proposals. I heard that it's possible to have one student per task from Google guys. So, I had considered breaking this task into pieces but I couldn't find good idea. You cannot have the students working as a group: thi http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2012/faqs#group As you know, this is a open source project. If you did already some work on here, Please keep an eye on this task. The transient GSoC experience does not represent much to your career. Making a contribution to the project you love or Acquiring ASF committership is more valuable.
        Hide
        Alejandro Claudio Esquivel Gonzalez added a comment -

        So, you say that the task has been already assigned and there will not be change on that? or that if you find a good idea to break this task into pieces you could accept more students?

        There are different formats to process the Sparse Matrix-vector multiplication if the Sparse matrix is in diagonal, tridiagonal form, band matrix, etc. I think you could identify the best format for each matrix and split the conversion and operation of different formats into different tasks.

        Also, there are more matrix operations that need different storage structures to be processed efficiently, for example editing a single value of the matrix works better if it's in Yale format.

        Thanks.

        Show
        Alejandro Claudio Esquivel Gonzalez added a comment - So, you say that the task has been already assigned and there will not be change on that? or that if you find a good idea to break this task into pieces you could accept more students? There are different formats to process the Sparse Matrix-vector multiplication if the Sparse matrix is in diagonal, tridiagonal form, band matrix, etc. I think you could identify the best format for each matrix and split the conversion and operation of different formats into different tasks. Also, there are more matrix operations that need different storage structures to be processed efficiently, for example editing a single value of the matrix works better if it's in Yale format. Thanks.
        Hide
        Thomas Jungblut added a comment -

        Hi Alejandro,

        the tasks are scored by the mentor and the person with the highest score gets the task, this will be sorted out this and next week.
        If you have already done research here then you should've let us know.
        Mikalai would have also got a higher score since he has contributed a few parts to Hama as well. Partitipation is weighted quite strongly, long term comittership as well.

        If you have ideas and motivation to help Hama with your research then feel free to participate. But we do not offer you five grands like Google. However the comittership in an uprising project is much more valuable than this amount of money, even for a student.

        Think about it.

        PS: to all others already applied to this task, I would be very proud if you want to help us out establishing a math package in Hama, we think this has huge potential in commodity cluster computing and contains lots of research and exciting tasks.

        Show
        Thomas Jungblut added a comment - Hi Alejandro, the tasks are scored by the mentor and the person with the highest score gets the task, this will be sorted out this and next week. If you have already done research here then you should've let us know. Mikalai would have also got a higher score since he has contributed a few parts to Hama as well. Partitipation is weighted quite strongly, long term comittership as well. If you have ideas and motivation to help Hama with your research then feel free to participate. But we do not offer you five grands like Google. However the comittership in an uprising project is much more valuable than this amount of money, even for a student. Think about it. PS: to all others already applied to this task, I would be very proud if you want to help us out establishing a math package in Hama, we think this has huge potential in commodity cluster computing and contains lots of research and exciting tasks.
        Hide
        Edward J. Yoon added a comment -

        Mikalai,
        Your name appears on my GSoC dashboard now. How is your work going?

        Show
        Edward J. Yoon added a comment - Mikalai, Your name appears on my GSoC dashboard now. How is your work going?
        Hide
        Mikalai Parafeniuk added a comment -

        Hello.
        Thanks for choosing me for this project. I am glad to announce that the work has started. First of all you can find some information about mine implementation timeline here: https://google-melange.appspot.com/gsoc/project/google/gsoc2012/mikalaj/18001

        Some ideas and questions about entire project:
        1) I propose to create separate package org.apache.hama.examples.matrix.sparse for this project and other code for matrix computing. I think it will be usefull
        because it will be hard to find other examples between mine classes.
        2) Whether to use mahout package for existing implementations of sparse matrices format or create my own implementation, based on their approach? I think it won't take much time to create formats and writables, and less dependencies will be brought to hama project.

        Now I am working on creating abstract interfaces for different formats of matrices. Here are some basic concepts:
        1) Create class MatrixCell. It will have next fields: row, column, value.
        2) Create interface MatrixFormat. The class will give possibility to return iterator for MatrixCell. Also format will support adding MatrixCell.
        3) Other custom formats like CRS and CCS should implement MatrixFormat.
        4) Create interface Converter for conversion between different formats.
        5) Create class BaseConverter which converts all formats without exploiting internal data structures: iterates MatixCell from one matrix and sets it to another.
        6) Other converters will implement converter and will exploit internal data structures of formats. To register some custom converters between two formats I will use configuration: it will consist of class name of input format, class name of output format, class name of converter.
        7) Use reflection in time of conversion. If custom converter for formats can be found, it will be used. Otherwise BaseConverter will be used .

        I think this approach gives independence of internal data format and gives opportunity to add data format quickly. Some questions, suggestions about this idea?

        Some general questions:
        1) I want to share snapshots of my code nearly once a week. But I think to release patches only when some piece of work is complete. It is a good idea to use git for this purpose?

        Show
        Mikalai Parafeniuk added a comment - Hello. Thanks for choosing me for this project. I am glad to announce that the work has started. First of all you can find some information about mine implementation timeline here: https://google-melange.appspot.com/gsoc/project/google/gsoc2012/mikalaj/18001 Some ideas and questions about entire project: 1) I propose to create separate package org.apache.hama.examples.matrix.sparse for this project and other code for matrix computing. I think it will be usefull because it will be hard to find other examples between mine classes. 2) Whether to use mahout package for existing implementations of sparse matrices format or create my own implementation, based on their approach? I think it won't take much time to create formats and writables, and less dependencies will be brought to hama project. Now I am working on creating abstract interfaces for different formats of matrices. Here are some basic concepts: 1) Create class MatrixCell. It will have next fields: row, column, value. 2) Create interface MatrixFormat. The class will give possibility to return iterator for MatrixCell. Also format will support adding MatrixCell. 3) Other custom formats like CRS and CCS should implement MatrixFormat. 4) Create interface Converter for conversion between different formats. 5) Create class BaseConverter which converts all formats without exploiting internal data structures: iterates MatixCell from one matrix and sets it to another. 6) Other converters will implement converter and will exploit internal data structures of formats. To register some custom converters between two formats I will use configuration: it will consist of class name of input format, class name of output format, class name of converter. 7) Use reflection in time of conversion. If custom converter for formats can be found, it will be used. Otherwise BaseConverter will be used . I think this approach gives independence of internal data format and gives opportunity to add data format quickly. Some questions, suggestions about this idea? Some general questions: 1) I want to share snapshots of my code nearly once a week. But I think to release patches only when some piece of work is complete. It is a good idea to use git for this purpose?
        Hide
        Edward J. Yoon added a comment -

        2) Whether to use mahout package for existing implementations of sparse matrices format or create my own implementation, based on their approach? I think it won't take much time to create formats and writables, and less dependencies will be brought to hama project.

        Please try to implement your own classes.

        Show
        Edward J. Yoon added a comment - 2) Whether to use mahout package for existing implementations of sparse matrices format or create my own implementation, based on their approach? I think it won't take much time to create formats and writables, and less dependencies will be brought to hama project. Please try to implement your own classes.
        Hide
        Edward J. Yoon added a comment -

        And, yes, git is good idea.

        Show
        Edward J. Yoon added a comment - And, yes, git is good idea.
        Hide
        Thomas Jungblut added a comment -

        Please try to implement your own classes.

        Would be cool to have some neat math classes for vectors and matrices though.

        You can have a look at https://github.com/thomasjungblut/tjungblut-math
        It is inspired by the class design of mahout-math.

        Here is the Writable implementation of the vectors: https://github.com/thomasjungblut/thomasjungblut-common/blob/master/src/de/jungblut/clustering/model/VectorWritable.java

        Just in case you need them, they are Apache2 considered, so you can just copy and paste to our project. But it would be cool if you add testcases.

        Show
        Thomas Jungblut added a comment - Please try to implement your own classes. Would be cool to have some neat math classes for vectors and matrices though. You can have a look at https://github.com/thomasjungblut/tjungblut-math It is inspired by the class design of mahout-math. Here is the Writable implementation of the vectors: https://github.com/thomasjungblut/thomasjungblut-common/blob/master/src/de/jungblut/clustering/model/VectorWritable.java Just in case you need them, they are Apache2 considered, so you can just copy and paste to our project. But it would be cool if you add testcases.
        Hide
        Mikalai Parafeniuk added a comment -

        Hello everybody. I am glad to say that i finished the most tedious activities
        in my university. So from now, implementation of this task is my priority.
        I am currently working on creating different matrix and vector formats.

        You can checkout current code from my github repository.
        https://github.com/ParafeniukMikalaj/gsoc-hama/
        Actually changes related to this task were placed in examples.linearalgebra package:
        https://github.com/ParafeniukMikalaj/gsoc-hama/tree/trunk/examples/src/main/java/org/apache/hama/examples/linearalgebra

        Recently I have created some abstract interfaces and classes for supporting different
        matrix formats. Could anyone make a review and give a feedback? I know that the amount of code is
        not to big, but I think the earlier I'll receive feedback, the earlier I'll correct the code.
        In the nearest future i plan to implement formats for dense matrices and vectors and one format
        for sparse matrices and vectors.
        Also I have a set of questions:
        1) Is using of reflection is a good idea to exploit internals of matrix format in time of convertion?
        Or it could be implemented simpler?
        2) I want to use HamaConfiguration class to register some custom converters between some
        matrix formats. Could anyone give me a starting point, where I can learn how Hadoop's configuration
        works?

        Show
        Mikalai Parafeniuk added a comment - Hello everybody. I am glad to say that i finished the most tedious activities in my university. So from now, implementation of this task is my priority. I am currently working on creating different matrix and vector formats. You can checkout current code from my github repository. https://github.com/ParafeniukMikalaj/gsoc-hama/ Actually changes related to this task were placed in examples.linearalgebra package: https://github.com/ParafeniukMikalaj/gsoc-hama/tree/trunk/examples/src/main/java/org/apache/hama/examples/linearalgebra Recently I have created some abstract interfaces and classes for supporting different matrix formats. Could anyone make a review and give a feedback? I know that the amount of code is not to big, but I think the earlier I'll receive feedback, the earlier I'll correct the code. In the nearest future i plan to implement formats for dense matrices and vectors and one format for sparse matrices and vectors. Also I have a set of questions: 1) Is using of reflection is a good idea to exploit internals of matrix format in time of convertion? Or it could be implemented simpler? 2) I want to use HamaConfiguration class to register some custom converters between some matrix formats. Could anyone give me a starting point, where I can learn how Hadoop's configuration works?
        Hide
        Thomas Jungblut added a comment - - edited

        Hi,

        had a bit of time so I give you a feedback:

        -Your abstract classes should not override methods in interfaces if they don't implement something.
        -You don't need to define methods in interfaces as abstract as they are implicitly
        -If you write your own read and write methods, then inherit from the Writable interface and implement it as well

        These are some minor things, so it won't affect your implementation too much.

        1) Is using of reflection is a good idea to exploit internals of matrix format in time of convertion? Or it could be implemented simpler?

        Do you have a concrete case where you want to use reflection?

        2) I want to use HamaConfiguration class to register some custom converters between some matrix formats. Could anyone give me a starting point, where I can learn how Hadoop's configuration works?

        I have given you the workflow of a configuration object here: https://issues.apache.org/jira/browse/HAMA-500?focusedCommentId=13253251&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13253251

        But basically it is some kind of Map<String,String> which holds information. It is bootstrapped by the usual hama configuration XMLs in classpath, then a client will modify it. Afterwards it will be serialized to XML and distributed through the cluster to be available on each task.

        If you want to register something, you can set the default in the configuration XML in hama's conf path and user can override it any time he wants to. Normally you would add helpers to your converter class that gets a configuration and some parameter that you want to set into it (See FileInputFormat.addInputPath for example).

        Show
        Thomas Jungblut added a comment - - edited Hi, had a bit of time so I give you a feedback: -Your abstract classes should not override methods in interfaces if they don't implement something. -You don't need to define methods in interfaces as abstract as they are implicitly -If you write your own read and write methods, then inherit from the Writable interface and implement it as well These are some minor things, so it won't affect your implementation too much. 1) Is using of reflection is a good idea to exploit internals of matrix format in time of convertion? Or it could be implemented simpler? Do you have a concrete case where you want to use reflection? 2) I want to use HamaConfiguration class to register some custom converters between some matrix formats. Could anyone give me a starting point, where I can learn how Hadoop's configuration works? I have given you the workflow of a configuration object here: https://issues.apache.org/jira/browse/HAMA-500?focusedCommentId=13253251&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13253251 But basically it is some kind of Map<String,String> which holds information. It is bootstrapped by the usual hama configuration XMLs in classpath, then a client will modify it. Afterwards it will be serialized to XML and distributed through the cluster to be available on each task. If you want to register something, you can set the default in the configuration XML in hama's conf path and user can override it any time he wants to. Normally you would add helpers to your converter class that gets a configuration and some parameter that you want to set into it (See FileInputFormat.addInputPath for example).
        Hide
        Mikalai Parafeniuk added a comment -

        Thanks for response, Thomas. Your comments were useful. I refactored code,
        and added classes for class conversion - HamaMatrixConverter, HamaVectorConverter.
        Code is available in my github repository. These classes are very unsafe right now,
        maybe some ideas how to correct this? Now I am going to implement some formats
        for matrices and vectors: dense, CRS, CCS. And after that I want to refactor generation
        of random matrices to use existing classes and create first package of unit tests.

        Show
        Mikalai Parafeniuk added a comment - Thanks for response, Thomas. Your comments were useful. I refactored code, and added classes for class conversion - HamaMatrixConverter, HamaVectorConverter. Code is available in my github repository. These classes are very unsafe right now, maybe some ideas how to correct this? Now I am going to implement some formats for matrices and vectors: dense, CRS, CCS. And after that I want to refactor generation of random matrices to use existing classes and create first package of unit tests.
        Hide
        Edward J. Yoon added a comment -

        It's so nice to see your progress and your process!

        Here's my minor comment:

            /**
             * Prints useful info about generated matrix: rows, columns, sparsity, items
             * count.
             */
            @Override
            public void cleanup(
                BSPPeer<NullWritable, NullWritable, Text, DoubleWritable, IntegerMessage> peer)
                throws IOException {
              if (peer.getPeerName().equals(masterTask)) {
                peer.write(new Text("Generated matix is"), null);
                peer.write(new Text("Sparsity    = "), new DoubleWritable(sparsity));
                peer.write(new Text("rows        = "), new DoubleWritable(rows));
                peer.write(new Text("columns     = "), new DoubleWritable(columns));
                peer.write(new Text("items count = "), new DoubleWritable(
                    generatedCount / 3));
                peer.write(new Text("\nValues:"), null);
                for (int i = 0; i < result.length; i++) {
                  peer.write(null, new DoubleWritable(result[i]));
                }
              }
            }
        

        If the purpose is just to print some information as a report, you can use "Counter" instead of writing information to a file.

        Great job!

        Show
        Edward J. Yoon added a comment - It's so nice to see your progress and your process! Here's my minor comment: /** * Prints useful info about generated matrix: rows, columns, sparsity, items * count. */ @Override public void cleanup( BSPPeer<NullWritable, NullWritable, Text, DoubleWritable, IntegerMessage> peer) throws IOException { if (peer.getPeerName().equals(masterTask)) { peer.write( new Text( "Generated matix is" ), null ); peer.write( new Text( "Sparsity = " ), new DoubleWritable(sparsity)); peer.write( new Text( "rows = " ), new DoubleWritable(rows)); peer.write( new Text( "columns = " ), new DoubleWritable(columns)); peer.write( new Text( "items count = " ), new DoubleWritable( generatedCount / 3)); peer.write( new Text( "\nValues:" ), null ); for ( int i = 0; i < result.length; i++) { peer.write( null , new DoubleWritable(result[i])); } } } If the purpose is just to print some information as a report, you can use "Counter" instead of writing information to a file. Great job!
        Hide
        Mikalai Parafeniuk added a comment -

        Thanks for comment, Edward, I will take it into account. I am glad to announce that first implementation of matrix formats with improved documentation can be found now in my github repository. I have implemented next matrix formats: Dense, Compressed Column Storage (CCS), Compressed Row Storage (CRS), Double Linked Format, Jagged Diagonal Storage (JDS). Also I have implemented next vector formats: Dense, Sparse. Maybe someone can propose some other useful matrix and vector formats to implement? In next few days I will be writing tests for created formats and converters and fixing bugs. Waiting for your reviews, ideas and comments.

        Show
        Mikalai Parafeniuk added a comment - Thanks for comment, Edward, I will take it into account. I am glad to announce that first implementation of matrix formats with improved documentation can be found now in my github repository. I have implemented next matrix formats: Dense, Compressed Column Storage (CCS), Compressed Row Storage (CRS), Double Linked Format, Jagged Diagonal Storage (JDS). Also I have implemented next vector formats: Dense, Sparse. Maybe someone can propose some other useful matrix and vector formats to implement? In next few days I will be writing tests for created formats and converters and fixing bugs. Waiting for your reviews, ideas and comments.
        Hide
        Thomas Jungblut added a comment -

        Cool, I've seen you using "init" methods, why do you prefer them over constructors? For me personally, I always forget to call them, so it might not always the best solution.

        Your sparse implementations are working with parallel lists, do you mind adding another RandomAccessSparse implementation with (hash-)maps?

        Show
        Thomas Jungblut added a comment - Cool, I've seen you using "init" methods, why do you prefer them over constructors? For me personally, I always forget to call them, so it might not always the best solution. Your sparse implementations are working with parallel lists, do you mind adding another RandomAccessSparse implementation with (hash-)maps?
        Hide
        Mikalai Parafeniuk added a comment -

        I thought that "init" method will be useful when user doesn't exactly knows matrix size in creation time (for example, reading matrix from file). But you are totally right, that it is easily to forget to call "init" method after matrix creation. So I created new constructor in AbstractMatrixFormat and AbstractVectorFormat classes. Constructor looks like this:

          public AbstractMatrixFormat(int rows, int columns) {
            this.rows = rows;
            this.columns = columns;
            init();
        }
        

        I think that RandomAccessSparse matrix format will be useful, so I added it. Refactored code with new format can be found in my github repository. Thanks for response, Thomas.

        Show
        Mikalai Parafeniuk added a comment - I thought that "init" method will be useful when user doesn't exactly knows matrix size in creation time (for example, reading matrix from file). But you are totally right, that it is easily to forget to call "init" method after matrix creation. So I created new constructor in AbstractMatrixFormat and AbstractVectorFormat classes. Constructor looks like this: public AbstractMatrixFormat( int rows, int columns) { this .rows = rows; this .columns = columns; init(); } I think that RandomAccessSparse matrix format will be useful, so I added it. Refactored code with new format can be found in my github repository. Thanks for response, Thomas.
        Hide
        Edward J. Yoon added a comment -

        I think, you have to start thinking about reordering and partitioning strategies. And also, I would recommend you to focus on Sparse matrix and vector formats.

        Show
        Edward J. Yoon added a comment - I think, you have to start thinking about reordering and partitioning strategies. And also, I would recommend you to focus on Sparse matrix and vector formats.
        Hide
        Edward J. Yoon added a comment - - edited

        Once it's roughly finished, we can improve code quality or optimize algorithm later. Don't think about code quality and technical practice at the moment.

        Show
        Edward J. Yoon added a comment - - edited Once it's roughly finished, we can improve code quality or optimize algorithm later. Don't think about code quality and technical practice at the moment.
        Hide
        Edward J. Yoon added a comment -

        Please try to use wiki. I've created one page for this issue

        http://wiki.apache.org/hama/SpMV

        If you need some help, please let me know. I'll help you.

        Show
        Edward J. Yoon added a comment - Please try to use wiki. I've created one page for this issue http://wiki.apache.org/hama/SpMV If you need some help, please let me know. I'll help you.
        Hide
        Mikalai Parafeniuk added a comment -

        Hello everybody. I have published some basic working examples with test cases of SpMV in my GitHub repositoty. But I have encountered some unclear problems. In debug mode in some configurations (testSpMVSmall for LAMathTest) I noticed that different peers are from different sides of barrier synchronization (peer.sync() method). For example I am getting NullPointerException while trying to access array of local matrices

        mLocal = mLocalArray[peerIndex];
        

        Initialization of array is made in setup method

        for (int i = 0; i < peerCount; i++) {
            mLocalArray[i] = strategy.getNewMatrixFormat();
            vLocalArray[i] = new SparseVector(m.getColumns());
        }
        

        I have placed additional peer.sync() in the end of setup method, but this doesn't helped. Now I will be working to avoid in-memory storage of matrices (thanks Apurv for comment), but I think this bug is crucial, and should be fixed as soon as possible. I think this occurs because I made some wrong initialization. Could anyone review my current code, especially test cases[1] and SpMV procedure[2] and help me to detect the problem?

        [1]http://goo.gl/HPMv6
        [2]http://goo.gl/gtwZ7

        Show
        Mikalai Parafeniuk added a comment - Hello everybody. I have published some basic working examples with test cases of SpMV in my GitHub repositoty. But I have encountered some unclear problems. In debug mode in some configurations (testSpMVSmall for LAMathTest) I noticed that different peers are from different sides of barrier synchronization (peer.sync() method). For example I am getting NullPointerException while trying to access array of local matrices mLocal = mLocalArray[peerIndex]; Initialization of array is made in setup method for ( int i = 0; i < peerCount; i++) { mLocalArray[i] = strategy.getNewMatrixFormat(); vLocalArray[i] = new SparseVector(m.getColumns()); } I have placed additional peer.sync() in the end of setup method, but this doesn't helped. Now I will be working to avoid in-memory storage of matrices (thanks Apurv for comment), but I think this bug is crucial, and should be fixed as soon as possible. I think this occurs because I made some wrong initialization. Could anyone review my current code, especially test cases [1] and SpMV procedure [2] and help me to detect the problem? [1] http://goo.gl/HPMv6 [2] http://goo.gl/gtwZ7
        Hide
        Edward J. Yoon added a comment -

        Can you provide readable pseudocode?

        Show
        Edward J. Yoon added a comment - Can you provide readable pseudocode?
        Hide
        Mikalai Parafeniuk added a comment -

        I think I should describe two methods: setup() and bsp() in SpMV class. Hama's initialization is made in method SparseMatrixVectorMultiplication and no pseudocode needed, code can be found in my GitHub repository.

        Variables: matrixLocalArray - array of local matrices, vectorLocalArray - array of local vectors, peerCount - number of peers, masterTask - name of master task, mLocal - local matrix, vLocal - local vector, peerName - name of current peer, inputMatrix and inputVector - input data.

        Setup procedure.

        masterTask = selectMasterTask();
        if (peerName == masterTask) {
          matrixLocalArray = createArrayOfMatrix();
          vectorLocalArray = createArrayOfVector();
          selectSpmvStrategy();
          selectSpmvMappers();
          for (int i = 0; i < peerCount; i++) {
            matrixLocalArray[i] = createEmptyMatrix()
            vectorLocalArray[i] = createEmptyVector();
          }
          for (matrixItem : inputMatrix) {
            int owner = getOwnerOfMatrixCell(matrixItem);
            matrixLocalArray[owner] = matrixItem;
          }
          for (vectorItem : inputVector) {
            int owner = getOwnerOfVectorCell(vectorItem);
            vectorLocalArray[owner] = vectorItem;
          }
        }
        sync();
        

        Bsp procedure.

        int index = definePeerIndex();
        mLocal = matrixLocalArray[index];
        vLocal = vectorLocalArray[index];
        makeOtherComputation();
        

        NullPointerException is thrown from time to time when i am trying to setup local matrix and array in bsp procedure, because different peers are from different sides of barrier synchronization. Thus masterTask haven't initialized cell in matrixLocalArray yeat and some peers trying to access local matrix in bsp method.

        Show
        Mikalai Parafeniuk added a comment - I think I should describe two methods: setup() and bsp() in SpMV class. Hama's initialization is made in method SparseMatrixVectorMultiplication and no pseudocode needed, code can be found in my GitHub repository. Variables: matrixLocalArray - array of local matrices, vectorLocalArray - array of local vectors, peerCount - number of peers, masterTask - name of master task, mLocal - local matrix, vLocal - local vector, peerName - name of current peer, inputMatrix and inputVector - input data. Setup procedure. masterTask = selectMasterTask(); if (peerName == masterTask) { matrixLocalArray = createArrayOfMatrix(); vectorLocalArray = createArrayOfVector(); selectSpmvStrategy(); selectSpmvMappers(); for ( int i = 0; i < peerCount; i++) { matrixLocalArray[i] = createEmptyMatrix() vectorLocalArray[i] = createEmptyVector(); } for (matrixItem : inputMatrix) { int owner = getOwnerOfMatrixCell(matrixItem); matrixLocalArray[owner] = matrixItem; } for (vectorItem : inputVector) { int owner = getOwnerOfVectorCell(vectorItem); vectorLocalArray[owner] = vectorItem; } } sync(); Bsp procedure. int index = definePeerIndex(); mLocal = matrixLocalArray[index]; vLocal = vectorLocalArray[index]; makeOtherComputation(); NullPointerException is thrown from time to time when i am trying to setup local matrix and array in bsp procedure, because different peers are from different sides of barrier synchronization. Thus masterTask haven't initialized cell in matrixLocalArray yeat and some peers trying to access local matrix in bsp method.
        Hide
        Edward J. Yoon added a comment -

        Hey Mikalai,

        I looked at your code closely today. Please delete all unnecessary classes and try to rewrite SpMV and SpMVTest programs.

          /**
           * Simple test.
           * multiplying
           *  [1 0 6 0]      [2]    [38]
           *  [0 4 0 0]  *   [3] =  [12]
           *  [0 2 3 0]      [6]    [24]
           *  [3 0 0 5]      [1]    [11]
           */
          //@Test
          public void testSpMVBasic() {
             Path m = generateRandomMatrix(); // 1. generate 4x4 matrix
             Path v = generateRandomMatrix(); // 2. generate 4x1 matrix (column vector)
        
             SpMV.main(m, v, TMP_OUT); // 2. job configuration
        
             verifyResult(TMP_OUT);
          }
        
        Show
        Edward J. Yoon added a comment - Hey Mikalai, I looked at your code closely today. Please delete all unnecessary classes and try to rewrite SpMV and SpMVTest programs. /** * Simple test. * multiplying * [1 0 6 0] [2] [38] * [0 4 0 0] * [3] = [12] * [0 2 3 0] [6] [24] * [3 0 0 5] [1] [11] */ //@Test public void testSpMVBasic() { Path m = generateRandomMatrix(); // 1. generate 4x4 matrix Path v = generateRandomMatrix(); // 2. generate 4x1 matrix (column vector) SpMV.main(m, v, TMP_OUT); // 2. job configuration verifyResult(TMP_OUT); }
        Hide
        Edward J. Yoon added a comment -

        Write the test data file to a local filesystem. Input should be a file, and your final output also should be a file.

        Show
        Edward J. Yoon added a comment - Write the test data file to a local filesystem. Input should be a file, and your final output also should be a file.
        Hide
        Edward J. Yoon added a comment -

        Mikalai, I looked at your new code but still strange to me.

        Here's my row-wise access style simple code.

          // Row-wise access style
          public static class RowWiseAccess
              extends
              BSP<IntWritable, ArrayWritable, IntWritable, IntWritable, BytesWritable> {
            private IntWritable[] columnVector = new IntWritable[4];
        
            /*
             * Let's assume
             *
             *  matrix A        x
             *                  1
             * 3  -2  0  4      2 
             * 1  5  -1  2   x  3
             * 0  4  3  1       4
             * 
             * @see org.apache.hama.bsp.BSP#bsp(org.apache.hama.bsp.BSPPeer)
             */
            @Override
            public void bsp(
                BSPPeer<IntWritable, ArrayWritable, IntWritable, IntWritable, BytesWritable> peer)
                throws IOException, SyncException, InterruptedException {
              KeyValuePair<IntWritable, ArrayWritable> row = null;
              // Read n-th row of matrix A
              while ((row = peer.readNext()) != null) {
                int key = row.getKey().get();
                int sum = 0;
                IntWritable[] rowVector = (IntWritable[]) row.getValue().get();
                // calculation of the scalar product of two vectors and n - 1 summation operations.
                for(int i = 0; i < 4; i++) {
                  sum += rowVector[i].get() * columnVector[i].get();
                }
                
                // So, complexity is O(n).
                peer.write(new IntWritable(key), new IntWritable(sum));
              }
            }
        
            @Override
            public void setup(
                BSPPeer<IntWritable, ArrayWritable, IntWritable, IntWritable, BytesWritable> peer)
                throws IOException, SyncException, InterruptedException {
              for (int i = 1; i < 5; i++) {
                columnVector[i - 1] = new IntWritable(i);
              }
            }
          }
        

        Then, As I mentioned you before, BSP framework automatically partitions the input data and executes each BSP task on that split.

        Matrix A                               x
        
        Task1 
        ---------------------------            
         3  -2  0  4                           1
         1  5  -1  2                           2
                                         x     3
        Task2                                  4
        ---------------------------
         0  4  3  1
        
        

        Can you understand this?

        Show
        Edward J. Yoon added a comment - Mikalai, I looked at your new code but still strange to me. Here's my row-wise access style simple code. // Row-wise access style public static class RowWiseAccess extends BSP<IntWritable, ArrayWritable, IntWritable, IntWritable, BytesWritable> { private IntWritable[] columnVector = new IntWritable[4]; /* * Let's assume * * matrix A x * 1 * 3 -2 0 4 2 * 1 5 -1 2 x 3 * 0 4 3 1 4 * * @see org.apache.hama.bsp.BSP#bsp(org.apache.hama.bsp.BSPPeer) */ @Override public void bsp( BSPPeer<IntWritable, ArrayWritable, IntWritable, IntWritable, BytesWritable> peer) throws IOException, SyncException, InterruptedException { KeyValuePair<IntWritable, ArrayWritable> row = null ; // Read n-th row of matrix A while ((row = peer.readNext()) != null ) { int key = row.getKey().get(); int sum = 0; IntWritable[] rowVector = (IntWritable[]) row.getValue().get(); // calculation of the scalar product of two vectors and n - 1 summation operations. for ( int i = 0; i < 4; i++) { sum += rowVector[i].get() * columnVector[i].get(); } // So, complexity is O(n). peer.write( new IntWritable(key), new IntWritable(sum)); } } @Override public void setup( BSPPeer<IntWritable, ArrayWritable, IntWritable, IntWritable, BytesWritable> peer) throws IOException, SyncException, InterruptedException { for ( int i = 1; i < 5; i++) { columnVector[i - 1] = new IntWritable(i); } } } Then, As I mentioned you before, BSP framework automatically partitions the input data and executes each BSP task on that split. Matrix A x Task1 --------------------------- 3 -2 0 4 1 1 5 -1 2 2 x 3 Task2 4 --------------------------- 0 4 3 1 Can you understand this?
        Hide
        Edward J. Yoon added a comment -

        CRS format can be also simply implemented by extending SequenceFile or MapFile as I commented above. Just write row index (K) and number of nonzero values (V). For example,

        out.writeInt(number of nonzero values);
        for(int i = 0; i < number of nonzero values; i++) {
          out.writeint(column index);
          out.writeint(value);
        }
        
        Show
        Edward J. Yoon added a comment - CRS format can be also simply implemented by extending SequenceFile or MapFile as I commented above. Just write row index (K) and number of nonzero values (V). For example, out.writeInt(number of nonzero values); for ( int i = 0; i < number of nonzero values; i++) { out.writeint(column index); out.writeint(value); }
        Hide
        Edward J. Yoon added a comment -

        Now do you understand why I said the load balance and partitioning are most important?

        Show
        Edward J. Yoon added a comment - Now do you understand why I said the load balance and partitioning are most important?
        Hide
        Edward J. Yoon added a comment -

        The reason of using framework's IO system is data locality.

        Show
        Edward J. Yoon added a comment - The reason of using framework's IO system is data locality.
        Hide
        Edward J. Yoon added a comment -

        Miklai,

        You can attach the patch to the JIRA. Under "More Actions" you should be able to attach a file.

        The patch looks good. I'll test this on my cluster and feedback for you.

        Show
        Edward J. Yoon added a comment - Miklai, You can attach the patch to the JIRA. Under "More Actions" you should be able to attach a file. The patch looks good. I'll test this on my cluster and feedback for you.
        Hide
        Edward J. Yoon added a comment -

        1. Don't forget the license headers and copyrights in each file.
        2. Please look at how ExampleDriver works. Also, it should able to run with command line arguments like other examples.
        3. The main() method is located at the bottom of the class (general convention).
        4. Please add usage guide. For example, Usage: spmv <input> <output> ...

        Please try to improve SpMV_v01.patch.

        I couldn't test today. One simple idea of load-balancing is row re-allocation using BSP communication API but let's see the performance of this algorithm first.

        Show
        Edward J. Yoon added a comment - 1. Don't forget the license headers and copyrights in each file. 2. Please look at how ExampleDriver works. Also, it should able to run with command line arguments like other examples. 3. The main() method is located at the bottom of the class (general convention). 4. Please add usage guide. For example, Usage: spmv <input> <output> ... Please try to improve SpMV_v01.patch. I couldn't test today. One simple idea of load-balancing is row re-allocation using BSP communication API but let's see the performance of this algorithm first.
        Hide
        Mikalai Parafeniuk added a comment -

        Hello.
        Edward, I saw you patch modification and agree with yours directory structure, replacement of ByteMessage to NullWritable, fix of not closed reader. So I decided not to submit my original patch. While waiting your feedback I will be learning how to create custom and efficient Partitioner.

        Show
        Mikalai Parafeniuk added a comment - Hello. Edward, I saw you patch modification and agree with yours directory structure, replacement of ByteMessage to NullWritable , fix of not closed reader. So I decided not to submit my original patch. While waiting your feedback I will be learning how to create custom and efficient Partitioner .
        Hide
        Mikalai Parafeniuk added a comment -

        Edward, I have tried to follow your tips, so as a result HAMA-524.patch appeared. I have also replaced my format of command line to format common to Hama.

        Show
        Mikalai Parafeniuk added a comment - Edward, I have tried to follow your tips, so as a result HAMA-524 .patch appeared. I have also replaced my format of command line to format common to Hama.
        Hide
        Edward J. Yoon added a comment -

        Looks good but it won't work on fully distributed mode. The reason (or how-to) is well described in http://wiki.apache.org/hama/DevelopBSP#Configure_variables_from_outside

        attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
        attempt_201207311829_0002_000141_0: java.lang.NullPointerException
        attempt_201207311829_0002_000141_0:     at org.apache.hama.examples.RandomMatrixGenerator.getSparsity(RandomMatrixGenerator.java:93)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.examples.RandomMatrixGenerator$MyGenerator.setup(RandomMatrixGenerator.java:149)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158)
        attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 INFO zookeeper.ZooKeeper: Session: 0x138dfcd2d9e00c4 closed
        attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 INFO zookeeper.ClientCnxn: EventThread shut down
        attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 ERROR bsp.BSPTask: Shutting down ping service.
        attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 FATAL bsp.GroomServer: Error running child
        attempt_201207311829_0002_000141_0: java.lang.NullPointerException
        attempt_201207311829_0002_000141_0:     at org.apache.hama.examples.RandomMatrixGenerator.getSparsity(RandomMatrixGenerator.java:93)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.examples.RandomMatrixGenerator$MyGenerator.setup(RandomMatrixGenerator.java:149)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158)
        attempt_201207311829_0002_000141_0: java.lang.NullPointerException
        attempt_201207311829_0002_000141_0:     at org.apache.hama.examples.RandomMatrixGenerator.getSparsity(RandomMatrixGenerator.java:93)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.examples.RandomMatrixGenerator$MyGenerator.setup(RandomMatrixGenerator.java:149)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
        attempt_201207311829_0002_000141_0:     at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158)
        12/07/31 18:32:07 INFO bsp.BSPJobClient: Job failed.
        
        Show
        Edward J. Yoon added a comment - Looks good but it won't work on fully distributed mode. The reason (or how-to) is well described in http://wiki.apache.org/hama/DevelopBSP#Configure_variables_from_outside attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 ERROR bsp.BSPTask: Error running bsp setup and bsp function. attempt_201207311829_0002_000141_0: java.lang.NullPointerException attempt_201207311829_0002_000141_0: at org.apache.hama.examples.RandomMatrixGenerator.getSparsity(RandomMatrixGenerator.java:93) attempt_201207311829_0002_000141_0: at org.apache.hama.examples.RandomMatrixGenerator$MyGenerator.setup(RandomMatrixGenerator.java:149) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158) attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 INFO zookeeper.ZooKeeper: Session: 0x138dfcd2d9e00c4 closed attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 INFO zookeeper.ClientCnxn: EventThread shut down attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 ERROR bsp.BSPTask: Shutting down ping service. attempt_201207311829_0002_000141_0: 12/08/01 01:28:03 FATAL bsp.GroomServer: Error running child attempt_201207311829_0002_000141_0: java.lang.NullPointerException attempt_201207311829_0002_000141_0: at org.apache.hama.examples.RandomMatrixGenerator.getSparsity(RandomMatrixGenerator.java:93) attempt_201207311829_0002_000141_0: at org.apache.hama.examples.RandomMatrixGenerator$MyGenerator.setup(RandomMatrixGenerator.java:149) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158) attempt_201207311829_0002_000141_0: java.lang.NullPointerException attempt_201207311829_0002_000141_0: at org.apache.hama.examples.RandomMatrixGenerator.getSparsity(RandomMatrixGenerator.java:93) attempt_201207311829_0002_000141_0: at org.apache.hama.examples.RandomMatrixGenerator$MyGenerator.setup(RandomMatrixGenerator.java:149) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143) attempt_201207311829_0002_000141_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158) 12/07/31 18:32:07 INFO bsp.BSPJobClient: Job failed.
        Hide
        Edward J. Yoon added a comment -

        You can set up your own cluster[1]. Please test yourself first and write a guide of this example on Wiki (like [2]).

        Then, "GSoC mission completed" and we can commit this to TRUNK.

        1. http://wiki.apache.org/hama/GettingStarted#Pseudo_Distributed_Mode
        2. http://wiki.apache.org/hama/PageRank

        Show
        Edward J. Yoon added a comment - You can set up your own cluster [1] . Please test yourself first and write a guide of this example on Wiki (like [2] ). Then, "GSoC mission completed" and we can commit this to TRUNK. 1. http://wiki.apache.org/hama/GettingStarted#Pseudo_Distributed_Mode 2. http://wiki.apache.org/hama/PageRank
        Hide
        Mikalai Parafeniuk added a comment -

        Hello everybody.
        I have tried to setup Hadoop and Hama cluster on my machine in pseudo-distributed mode. I have set up Hadoop cluster and launched WordCount example successfully. After that I tried to setup Hama in pseudo-distributed mode. When I launched start-bspd.sh output was like in tutorial, but jps command didn't show BspMasterRunner. I explored logs and found next line: "java.io.IOException: Broken pipe". Could someone help me to figure this issue out? I used tutorials from here[1] and here[2]. I also attached archive with logs of hama and main configuration files.

        [1]http://wiki.apache.org/hama/GettingStarted
        [2]http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#running-a-mapreduce-job

        Show
        Mikalai Parafeniuk added a comment - Hello everybody. I have tried to setup Hadoop and Hama cluster on my machine in pseudo-distributed mode. I have set up Hadoop cluster and launched WordCount example successfully. After that I tried to setup Hama in pseudo-distributed mode. When I launched start-bspd.sh output was like in tutorial, but jps command didn't show BspMasterRunner. I explored logs and found next line: "java.io.IOException: Broken pipe". Could someone help me to figure this issue out? I used tutorials from here [1] and here [2] . I also attached archive with logs of hama and main configuration files. [1] http://wiki.apache.org/hama/GettingStarted [2] http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#running-a-mapreduce-job
        Hide
        Mikalai Parafeniuk added a comment -

        logs and configuration files.

        Show
        Mikalai Parafeniuk added a comment - logs and configuration files.
        Hide
        Suraj Menon added a comment -

        Hi Mikalai, your configuration files seemed correct to me.(They worked fine for me). From the logs I could find that it failed because it could not initialize the filesystem. Any chance that you are running HDFS and BSP processes as different users? I shall check the config files as soon as I find time.

        Show
        Suraj Menon added a comment - Hi Mikalai, your configuration files seemed correct to me.(They worked fine for me). From the logs I could find that it failed because it could not initialize the filesystem. Any chance that you are running HDFS and BSP processes as different users? I shall check the config files as soon as I find time.
        Hide
        Mikalai Parafeniuk added a comment -

        Hi Suraj. I checked that I am running hdfs and hama from the same user one more time. Now I am getting EOFException. Also I have found some suspicious lines in log files.
        Hadoop NameNode log:

        2012-08-06 15:12:01,283 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 127.0.0.1:46588 got version 4 expected version 3
        

        Hama BspMaster log:

        2012-08-06 15:12:01,297 ERROR org.apache.hama.bsp.BSPMaster: Can't get connection to Hadoop Namenode!
        java.io.IOException: Call to localhost/127.0.0.1:54310 failed on local exception: java.io.EOFException
        

        Maybe the problem is that I have installed incompatible versions of Hadoop and Hama? I attached file with typescript of my terminal session and hadoop and hama log files. Typescript can be viewed in readable version with help of cat command. Configuration is the same, as in previous archive.

        Show
        Mikalai Parafeniuk added a comment - Hi Suraj. I checked that I am running hdfs and hama from the same user one more time. Now I am getting EOFException. Also I have found some suspicious lines in log files. Hadoop NameNode log: 2012-08-06 15:12:01,283 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 127.0.0.1:46588 got version 4 expected version 3 Hama BspMaster log: 2012-08-06 15:12:01,297 ERROR org.apache.hama.bsp.BSPMaster: Can't get connection to Hadoop Namenode! java.io.IOException: Call to localhost/127.0.0.1:54310 failed on local exception: java.io.EOFException Maybe the problem is that I have installed incompatible versions of Hadoop and Hama? I attached file with typescript of my terminal session and hadoop and hama log files. Typescript can be viewed in readable version with help of cat command. Configuration is the same, as in previous archive.
        Hide
        Suraj Menon added a comment -

        Your logs suggested that you are using older version of HDFS. I hope you are running your changes on Hama-0.5 with HDFS 1.0 such that all the jars and config files are consistent across your cluster. The last section of the installation document you followed has steps to upgrade HDFS.

        Show
        Suraj Menon added a comment - Your logs suggested that you are using older version of HDFS. I hope you are running your changes on Hama-0.5 with HDFS 1.0 such that all the jars and config files are consistent across your cluster. The last section of the installation document you followed has steps to upgrade HDFS.
        Hide
        Mikalai Parafeniuk added a comment -

        Thanks for help, Suraj and Apurv. I have installed hadoop-1.0.3 instead of older version and now Hama is working fine in pseudo-distributed mode.

        Show
        Mikalai Parafeniuk added a comment - Thanks for help, Suraj and Apurv. I have installed hadoop-1.0.3 instead of older version and now Hama is working fine in pseudo-distributed mode.
        Hide
        Mikalai Parafeniuk added a comment -

        Hello everyone.
        I changed my patch to work pseudo-distributed mode. Now SpMV doesn't throws NullPointerException, but some bugs are still remain. I think, it is a time to show new version of patch. I have also changed my GitHub repository for SpMV project[1] - now it contains only files related to SpMV. Also I edited wiki page for spmv[2]. Any feedback is welcome.

        [1]https://github.com/ParafeniukMikalaj/spmv
        [2]http://wiki.apache.org/hama/SpMV#Implementation

        Show
        Mikalai Parafeniuk added a comment - Hello everyone. I changed my patch to work pseudo-distributed mode. Now SpMV doesn't throws NullPointerException , but some bugs are still remain. I think, it is a time to show new version of patch. I have also changed my GitHub repository for SpMV project [1] - now it contains only files related to SpMV. Also I edited wiki page for spmv [2] . Any feedback is welcome. [1] https://github.com/ParafeniukMikalaj/spmv [2] http://wiki.apache.org/hama/SpMV#Implementation
        Hide
        Mikalai Parafeniuk added a comment -

        Hello everyone. I am glad to say, that this version of SpMV works stable on my machine. I submitted new version of patch, updated github repository with SpMV, updated wiki page for SpMV. I tested this version in pseudo-distributed mode with different problem sizes (4x4, 10x10, 100x100, 1000x1000). Firm pencils down date for gsoc is scheduled for monday, so any feedback about code or wiki page, advises will be highly appreciated.
        Also I have one little problem : when I launched SpMV in pair with RandomMatrixGenerator for problem size 10000x10000 on local machine I run out of memory. After that I cleaned up directories, mentioned in config files and tmp, but there is still little free memory. Where can I found internal Hadoop and Hama data to free some space?

        Show
        Mikalai Parafeniuk added a comment - Hello everyone. I am glad to say, that this version of SpMV works stable on my machine. I submitted new version of patch, updated github repository with SpMV, updated wiki page for SpMV. I tested this version in pseudo-distributed mode with different problem sizes (4x4, 10x10, 100x100, 1000x1000). Firm pencils down date for gsoc is scheduled for monday, so any feedback about code or wiki page, advises will be highly appreciated. Also I have one little problem : when I launched SpMV in pair with RandomMatrixGenerator for problem size 10000x10000 on local machine I run out of memory. After that I cleaned up directories, mentioned in config files and tmp, but there is still little free memory. Where can I found internal Hadoop and Hama data to free some space?
        Hide
        Edward J. Yoon added a comment -

        Good job, Mikalai. Let's commit your project code.

        And, do you think you can improve this later?

        Show
        Edward J. Yoon added a comment - Good job, Mikalai. Let's commit your project code. And, do you think you can improve this later?
        Hide
        Mikalai Parafeniuk added a comment -

        I think i can improve patch, as we told earlier. As described on wike page, we can improve partitioning. I think to use some ideas from paper on spmv from Stanford. Despite of results of GSoC, I liked work in Hama community and I am planning to continue it after program ends.

        Show
        Mikalai Parafeniuk added a comment - I think i can improve patch, as we told earlier. As described on wike page, we can improve partitioning. I think to use some ideas from paper on spmv from Stanford. Despite of results of GSoC, I liked work in Hama community and I am planning to continue it after program ends.
        Hide
        Thomas Jungblut added a comment -

        Hi Mikalai, since we want to commit this soon, I give you a little feedback.

        • In WritableUtil you could define the methods as static (actually you should because this is a utility class). also if you have a log, don't output to STDOUT, this will vanish in distributed mode to /dev/null.
        • By convention (do we have a common sense here?) I name all the BSP's with the suffix BSP, e.G. KMeansBSP. Maybe (SpMV-)Executor is not a so good suffix here.
        • In SpMVExecutor there is a not needed double cast on line 170, also in the random gen arround line 204.

        Also I see a few performance hits, but I would rather profile it than guessing arround it. Will take a look at bigger inputs next time and profile with yourkit.

        In general, we should try to consider moving it to ml module, so it can share the math stuff and writables so we can move some stuff out of example utils.

        In the end, very clean code, well formatted and documented. Thanks Mikalai, great contribution.

        Show
        Thomas Jungblut added a comment - Hi Mikalai, since we want to commit this soon, I give you a little feedback. In WritableUtil you could define the methods as static (actually you should because this is a utility class). also if you have a log, don't output to STDOUT, this will vanish in distributed mode to /dev/null. By convention (do we have a common sense here?) I name all the BSP's with the suffix BSP, e.G. KMeansBSP. Maybe (SpMV-)Executor is not a so good suffix here. In SpMVExecutor there is a not needed double cast on line 170, also in the random gen arround line 204. Also I see a few performance hits, but I would rather profile it than guessing arround it. Will take a look at bigger inputs next time and profile with yourkit. In general, we should try to consider moving it to ml module, so it can share the math stuff and writables so we can move some stuff out of example utils. In the end, very clean code, well formatted and documented. Thanks Mikalai, great contribution.
        Hide
        Mikalai Parafeniuk added a comment -

        Hello everyone. I took into account comments of Thomas, and created a new version of patch - HAMA_524_v3.patch.

        I want to know opinion of other developers about moving SpMV to ml package. I think this movement is valuable, because after it we will have a single set of classes for vectors and matrices formats and writables.

        Question about GSoC. A program has finished and I am required to submit code sample to google's repository. Can I submit my patch? Or something else is needed?

        Show
        Mikalai Parafeniuk added a comment - Hello everyone. I took into account comments of Thomas, and created a new version of patch - HAMA_524_v3.patch. I want to know opinion of other developers about moving SpMV to ml package. I think this movement is valuable, because after it we will have a single set of classes for vectors and matrices formats and writables. Question about GSoC. A program has finished and I am required to submit code sample to google's repository. Can I submit my patch? Or something else is needed?
        Hide
        Thomas Jungblut added a comment -

        Thanks!

        Can I submit my patch? Or something else is needed?

        Last year just uploading the patch was totally fine.

        Show
        Thomas Jungblut added a comment - Thanks! Can I submit my patch? Or something else is needed? Last year just uploading the patch was totally fine.
        Hide
        Tommaso Teofili added a comment -

        I agree this is a great contribution.

        I want to know opinion of other developers about moving SpMV to ml package.

        yep, +1 for moving it there, we may eventually re abstract those math related code to an even more generic module (e.g. hama-math).

        Show
        Tommaso Teofili added a comment - I agree this is a great contribution. I want to know opinion of other developers about moving SpMV to ml package. yep, +1 for moving it there, we may eventually re abstract those math related code to an even more generic module (e.g. hama-math).
        Hide
        Thomas Jungblut added a comment -

        So what are we going to do with this?

        Can we put this into trunk, along with HAMA-500?

        Show
        Thomas Jungblut added a comment - So what are we going to do with this? Can we put this into trunk, along with HAMA-500 ?
        Hide
        Mikalai Parafeniuk added a comment -

        I had some rest for past two weeks. I am going to move this patch to ml package in few days. So we will be able to pick better patch.

        Show
        Mikalai Parafeniuk added a comment - I had some rest for past two weeks. I am going to move this patch to ml package in few days. So we will be able to pick better patch.
        Hide
        Thomas Jungblut added a comment -

        Perfect, thank you very much.

        Show
        Thomas Jungblut added a comment - Perfect, thank you very much.
        Hide
        Edward J. Yoon added a comment -

        > release is coming soon?

        was wanted to put this to 0.6 release. But NO need to hurry MiKalai.

        Show
        Edward J. Yoon added a comment - > release is coming soon? was wanted to put this to 0.6 release. But NO need to hurry MiKalai.
        Hide
        Edward J. Yoon added a comment -

        Committed. Thanks Mikalai.

        Show
        Edward J. Yoon added a comment - Committed. Thanks Mikalai.

          People

          • Assignee:
            Mikalai Parafeniuk
            Reporter:
            Edward J. Yoon
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development