[SPARK-15575] Remove breeze from dependencies?

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 26/May/16 18:20
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

This JIRA is for discussing whether we should remove Breeze from the dependencies of MLlib. The main issues with Breeze are Scala 2.12 support and performance issues.

There are a few paths:

1. Keep dependency. This could be OK, especially if the Scala version issues are fixed within Breeze.
2. Remove dependency
   1. Implement our own linear algebra operators as needed
   2. Design a way to build Spark using custom linalg libraries of the user's choice. E.g., you could build MLlib using Breeze, or any other library supporting the required operations. This might require significant work. See SPARK-6442 for related discussion.

Issue Links

• is related to: SPARK-6442 MLlib Local Linear Algebra Package
• relates to: SPARK-15882 Discuss distributed linear algebra in spark.ml package