[SPARK-14450] Python OneVsRest should train multiple models at once

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: ML, PySpark
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 07/Apr/16 02:24
• Updated: 12/Jun/17 23:55
• Resolved: 12/Jun/17 23:55

Description

SPARK-7861 adds a Python wrapper for OneVsRest. Because of possible issues related to using existing libraries like multiprocessing, we are not training multiple models in parallel initially.

This issue is for prototyping, testing, and implementing a way to train multiple models at once. Speaking with joshrosen, a good option might be the concurrent.futures package:

• Python 3.x: https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures
• Python 2.x: https://pypi.python.org/pypi/futures

We will not add this for Spark 2.0, but it will be good to investigate for 2.1.

Issue Links

• is duplicated by: SPARK-21027 Parallel One vs. Rest Classifier
• is related to: SPARK-7861 Python wrapper for OneVsRest