Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13944

Separate out local linear algebra as a standalone module without Spark dependency

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Build, ML
    • Labels:
      None
    • Target Version/s:

      Description

      Separate out linear algebra as a standalone module without Spark dependency to simplify production deployment. We can call the new module mllib-local, which might contain local models in the future.

      The major issue is to remove dependencies on user-defined types.

      The package name will be changed from mllib to ml. For example, Vector will be changed from `org.apache.spark.mllib.linalg.Vector` to `org.apache.spark.ml.linalg.Vector`. The return vector type in the new ML pipeline will be the one in ML package; however, the existing mllib code will not be touched. As a result, this will potentially break the API. Also, when the vector is loaded from mllib vector by Spark SQL, the vector will automatically converted into the one in ml package.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dbtsai DB Tsai
                Reporter:
                mengxr Xiangrui Meng
                Shepherd:
                Xiangrui Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: