Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35295

Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: GraphX, ML, MLlib
    • Labels:
      None

      Description

      As discussed in https://github.com/apache/spark/pull/32253, Spark cannot distribute `com.github.fommil.netlib:all` due to licensing reasons. This limits the ability of Spark to take advantage out of the box of native libraries that do provide hardware acceleration (SIMD and GPUs).

      With `dev.ludovic.netlib:2.0.0`, it is possible to take advantage of native libraries, but without the licensing limitations, since it doesn't link nor distribute any GPL or LGPL libraries.

      That would allow any other of Spark to install a native library like OpenBLAS or Intel MKL and take advantage transparently of these libraries.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                luhenry Ludovic Henry
                Reporter:
                luhenry Ludovic Henry
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: