[SPARK-7536] Audit MLlib Python API for 1.4

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.5.0
• Component/s: MLlib, PySpark
• Target Version/s: 1.5.0
• Assignee: Yanbo Liang
• Reporter: Joseph K. Bradley
• Created: 11/May/15 19:33
• Updated: 21/Aug/15 03:16
• Resolved: 10/Aug/15 18:04

Description

*NOTE: This is targeted at 1.5.0 because it has so many useful links for JIRAs targeted for 1.5.0. In the future, we should create a new JIRA for linking future items.*

For new public APIs added to MLlib, we need to check the generated HTML doc and compare the Scala & Python versions. We need to track:

• Inconsistency: Do class/method/parameter names match? SPARK-7667
• Docs: Is the Python doc missing or just a stub? We want the Python doc to be as complete as the Scala doc. SPARK-7666, SPARK-6173
• API breaking changes: These should be very rare but are occasionally either necessary (intentional) or accidental. These must be recorded and added in the Migration Guide for this release. SPARK-7665
  • Note: If the API change is for an Alpha/Experimental/DeveloperApi component, please note that as well.
• Missing classes/methods/parameters: We should create to-do JIRAs for functionality missing from Python.
  • classification
    • StreamingLogisticRegressionWithSGD SPARK-7633
  • clustering
    • GaussianMixture SPARK-6258
    • LDA SPARK-6259
    • Power Iteration Clustering SPARK-5962
    • StreamingKMeans SPARK-4118
  • evaluation
    • MultilabelMetrics SPARK-6094
  • feature
    • ElementwiseProduct SPARK-7605
    • PCA SPARK-7604
  • linalg
    • Distributed linear algebra SPARK-6100
  • pmml.export SPARK-7638
  • regression
    • StreamingLinearRegressionWithSGD SPARK-4127
  • stat
    • KernelDensity SPARK-7639
  • util
    • MLUtils SPARK-6263

Issue Links

• contains: SPARK-4118 Create python bindings for Streaming KMeans
• contains: SPARK-4127 Streaming Linear Regression- Python bindings
• contains: SPARK-6094 Add MultilabelMetrics in PySpark/MLlib
• contains: SPARK-6258 Python MLlib API missing items: Clustering
• contains: SPARK-6263 Python MLlib API missing items: Utils
• contains: SPARK-7633 Streaming Logistic Regression- Python bindings
• contains: SPARK-5962 [MLLIB] Python support for Power Iteration Clustering
• contains: SPARK-7604 Python API for PCA and PCAModel
• contains: SPARK-7605 Python API for ElementwiseProduct
• contains: SPARK-7639 Add Python API for Statistics.kernelDensity
• contains: SPARK-7638 Python API for pmml.export
• contains: SPARK-6259 Python API for LDA
• contains: SPARK-9122 spark.mllib regression should support batch predict
• contains: SPARK-8068 Add confusionMatrix method at class MulticlassMetrics in pyspark/mllib
• contains: SPARK-7667 MLlib Python API consistency check
• contains: SPARK-7203 Python API for local linear algebra
• contains: SPARK-3258 Python API for streaming MLlib algorithms
• contains: SPARK-5694 Python API for evaluation metrics
• contains: SPARK-6100 Distributed linear algebra in PySpark/MLlib
• contains: SPARK-6173 Python doc parity with Scala/Java in MLlib
• contains: SPARK-7665 MLlib Python API breaking changes check between 1.3 & 1.4
• contains: SPARK-7666 MLlib Python doc parity check
• contains: SPARK-8757 Check missing and add user guide for MLlib Python API