Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17847

Reduce shuffled data size of GaussianMixture & copy the implementation from mllib to ml

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: ML, MLlib
    • Labels:
      None
    • Target Version/s:

      Description

      Copy GaussianMixture implementation from mllib to ml, then we can add new features to it.
      I left mllib GaussianMixture untouched, unlike some other algorithms to wrap the ml implementation. For the following reasons:

      • mllib GaussianMixture allow k == 1, but ml does not.
      • mllib GaussianMixture supports setting initial model, but ml does not support currently. (We will definitely add this feature for ml in the future)

      Meanwhile, There is a big performance improvement for GaussianMixture in this task. Since the covariance matrix of multivariate gaussian distribution is symmetric, we can only store the upper triangular part of the matrix and it will greatly reduce the shuffled data size.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yanboliang Yanbo Liang
                Reporter:
                yanboliang Yanbo Liang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: