[SPARK-5814] Remove JBLAS from runtime dependencies

• Type: Dependency upgrade
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: GraphX, MLlib
• Target Version/s: 1.4.0
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 14/Feb/15 01:04
• Updated: 12/Mar/15 08:39
• Resolved: 12/Mar/15 08:39

Description

We are using mixed breeze/netlib-java and jblas code in MLlib. They take different approaches to utilize native libraries and we should keep only one of them. netlib-java has a clear separation between Java implementation and native JNI libraries, while JBLAS packs statically linked binaries that causes license issues (SPARK-5669). So we want to remove JBLAS from Spark runtime.

One issue with this approach is that we have JBLAS' DoubleMatrix exposed (by mistake) in SVDPlusPlus of GraphX. We should deprecate it and replace `DoubleMatrix` by `Array[Double]`.

Issue Links

• is related to: SPARK-5669 Spark assembly includes incompatibly licensed libgfortran, libgcc code via JBLAS
• is related to: SPARK-5815 Deprecate SVDPlusPlus APIs that expose DoubleMatrix from JBLAS
• links to: [Github] Pull Request #4699 (mengxr)