Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4591

Algorithm/model parity for spark.ml (Scala)

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Critical
    • Resolution: Done
    • None
    • None
    • ML
    • None

    Description

      This is an umbrella JIRA for porting spark.mllib implementations to use the DataFrame-based API defined under spark.ml. We want to achieve critical feature parity for the next release.

      Instructions for 3 subtask types

      Review tasks: detailed review of a subpackage to identify feature gaps between spark.mllib and spark.ml.

      • Should be listed as a subtask of this umbrella.
      • Review subtasks cover major algorithm groups. To pick up a review subtask, please:
        • Comment that you are working on it.
        • Compare the public APIs of spark.ml vs. spark.mllib.
        • Comment on all missing items within spark.ml: algorithms, models, methods, features, etc.
        • Check for existing JIRAs covering those items. If there is no existing JIRA, create one, and link it to your comment.

      Critical tasks: higher priority missing features which are required for this umbrella JIRA.

      • Should be linked as "requires" links.

      Other tasks: lower priority missing features which can be completed after the critical tasks.

      • Should be linked as "contains" links.

      Excluded items

      This does not include:

      • Python: We can compare Scala vs. Python in spark.ml itself.
      • Moving linalg to spark.ml: SPARK-13944
      • Streaming ML: Requires stabilizing some internal APIs of structured streaming first

      TODO list

      Critical issues

      Lower priority issues

      • Missing methods within algorithms (see Issue Links below)
      • evaluation submodule
      • stat submodule (should probably be covered in DataFrames)
      • Developer-facing submodules:
        • optimization (including SPARK-17136)
        • random, rdd
        • util

      To be prioritized

      Attachments

        Issue Links

          contains

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-13025 Allow user to specify the initial model when training LogisticRegression

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-14712 spark.ml LogisticRegressionModel.toString should summarize model

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          depends upon

          Sub-task - The sub-task of the issue SPARK-3702 Standardize MLlib classes for learners, models

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          requires

          New Feature - A new feature of the product, which has yet to be developed. SPARK-14709 spark.ml API for linear SVM

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-15784 Add Power Iteration Clustering to spark.ml

          • Major - Major loss of function.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-14501 spark.ml parity for fpm - frequent items

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              mengxr Xiangrui Meng
              Votes:
              4 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: