Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13944

Separate out local linear algebra as a standalone module without Spark dependency

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • Build, ML
    • None

    Description

      Separate out linear algebra as a standalone module without Spark dependency to simplify production deployment. We can call the new module mllib-local, which might contain local models in the future.

      The major issue is to remove dependencies on user-defined types.

      The package name will be changed from mllib to ml. For example, Vector will be changed from `org.apache.spark.mllib.linalg.Vector` to `org.apache.spark.ml.linalg.Vector`. The return vector type in the new ML pipeline will be the one in ML package; however, the existing mllib code will not be touched. As a result, this will potentially break the API. Also, when the vector is loaded from mllib vector by Spark SQL, the vector will automatically converted into the one in ml package.

      Attachments

        Issue Links

          Activity

            People

              dbtsai DB Tsai
              mengxr Xiangrui Meng
              Xiangrui Meng Xiangrui Meng
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: