Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1786

Make classes implements Serializable for Spark 1.5+

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.13.2
    • Component/s: Math
    • Labels:

      Description

      Spark 1.5 comes with a new very efficient serializer that uses code generation. It is twice as fast as kryo. When using mahout, we have to set KryoSerializer because some classes aren't serializable otherwise.

      I suggest to declare Math classes as "implements Serializable" where needed. For instance, to use coocurence package in spark 1.5, we had to modify AbstractMatrix, AbstractVector, DenseVector and SparseRowMatrix to make it work without Kryo.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pferrel Pat Ferrel
                Reporter:
                FlamingMike Michel Lemay
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m