Description
As discussed in https://github.com/apache/spark/pull/32253, Spark cannot distribute `com.github.fommil.netlib:all` due to licensing reasons. This limits the ability of Spark to take advantage out of the box of native libraries that do provide hardware acceleration (SIMD and GPUs).
With `dev.ludovic.netlib:2.0.0`, it is possible to take advantage of native libraries, but without the licensing limitations, since it doesn't link nor distribute any GPL or LGPL libraries.
That would allow any other of Spark to install a native library like OpenBLAS or Intel MKL and take advantage transparently of these libraries.
Attachments
Issue Links
- relates to
-
SPARK-35392 Flaky Test: GaussianMixture at ml/clustering.py and Word2Vec at ml/feature.py
-
- Resolved
-
- links to
