[SPARK-14706] Python ML persistence integration test

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML, PySpark
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 18/Apr/16 16:59
• Updated: 21/May/19 04:16
• Resolved: 21/May/19 04:16

Description

Goal: extend integration test in ml/tests.py.

In the PersistenceTest suite, there is a method _compare_pipelines. This issue includes:

• Extending _compare_pipelines to handle CrossValidator, TrainValidationSplit, and OneVsRest
• Adding an integration test in PersistenceTest which includes nested meta-algorithms. E.g.: Pipeline[ CrossValidator[ TrainValidationSplit[ OneVsRest[ LogisticRegression ] ] ] ].

Issue Links

• relates to: SPARK-14924 Tuning estimatorParamMaps with OneVsRest.classifier fails during persistence
• relates to: SPARK-14931 Mismatched default Param values between pipelines in Spark and PySpark
• relates to: SPARK-14973 The CrossValidator and TrainValidationSplit miss the seed when saving and loading
• relates to: SPARK-14771 Python ML Param and UID issues
• requires: SPARK-11939 PySpark support model export/import for Pipeline API
• links to: [Github] Pull Request #12604 (yinxusen)

Sub-Tasks

• Python ML persistence integration test: OneVsRest: Resolved