[SPARK-15944] Make spark.ml package backward compatible with spark.mllib vectors

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: ML, MLlib
• Target Version/s: 2.1.0
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 14/Jun/16 15:14
• Updated: 02/Nov/16 00:14
• Resolved: 02/Nov/16 00:14

Description

During QA, we found that it is not trivial to convert a DataFrame with old vector columns to new vector columns. So it would be easier for users to migrate their datasets and pipelines if we:

1) provide utils to convert DataFrames with vector columns
2) automatically detect and convert old vector columns in ML pipelines

This is an umbrella JIRA to track the progress.

Issue Links

• contains: SPARK-15986 SVMWithSGD requires LabeledPoint of Regression

Sub-Tasks

• Implement conversion utils in Scala/Java: Resolved
• Wrap the conversion utils in Python: Resolved
• Make pipeline components backward compatible with old vector columns in Scala/Java: Closed
• Make pipeline components backward compatible with old vector columns in Python: Closed
• Make model loading backward compatible with saved models using old vector columns: Resolved
• Implement util method for ML Matrix conversion in scala/java: Resolved
• Wrap the Matrix conversion utils in Python: Resolved
• Implement conversion utility functions for single instances in Python: Resolved