[SPARK-7610] Design clustering abstractions for Pipelines API

• Type: Brainstorming
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 13/May/15 18:50
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

We will soon start adding clustering algorithms to the Pipelines API. We should discuss what abstractions we should define for clustering algorithms:

• Are there standard APIs for clustering?
• What clustering algorithms might not fit in the normal APIs?
• How can the APIs benefit (a) users and (b) developers?

W.r.t. benefitting users and developers, I'm envisioning:

• traits for users
  • These should help standardize the API, but we'll have to plan ahead carefully.
• abstract classes with some boilerplate code pre-implemented for developers (similar to the current Prediction developer APIs)
  • These will be available if helpful, but developers should not need to use them.

Issue Links

• Is contained by: SPARK-10817 ML abstraction umbrella
• relates to: SPARK-7879 KMeans API for spark.ml Pipelines
• relates to: SPARK-5565 LDA wrapper for spark.ml package