[PIO-116] PySpark Support

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0-incubating
• Component/s: Core
• Assignee: Shinsuke Sugaya
• Reporter: Shinsuke Sugaya
• Created: 29/Aug/17 09:17
• Updated: 05/Dec/17 22:44
• Resolved: 12/Sep/17 16:34

Description

This provides PySpark support with minimum PIO changes.

1. Support pyspark on pio-shell
2. Add python files to use pyspark
3. Add --main-py-file option to "pio train" to submit .py file to spark

Note that this provides only fixes for Spark 2.x.
(because this fixes expect to use SparkML)

Sample project is:
https://github.com/jpioug/predictionio-template-iris
(For prediction API, Scala code is used.)

Issue Links

• links to: GitHub Pull Request #427