Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21688

performance improvement in mllib SVM with native BLAS

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: MLlib
    • Labels:
      None
    • Environment:

      4 nodes: 1 master node, 3 worker nodes
      model name : Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
      Memory : 180G
      num of core per node: 10

      Description

      in current mllib SVM implementation, we found that the CPU is not fully utilized, one reason is that f2j blas is set to be used in the HingeGradient computation. As we found out earlier (https://issues.apache.org/jira/browse/SPARK-21305) that with proper settings, native blas is generally better than f2j on the uni-test level, here we make the blas operations in SVM go with MKL blas and get an end to end performance report showing that in most cases native blas outperformance f2j blas up to 50%.
      So, we suggest removing those f2j-fixed calling and going for native blas if available. If this proposal is acceptable, we will move on to benchmark other algorithms impacted.

        Attachments

        1. native-trywait.png
          416 kB
          Vincent
        2. ddot unitest.png
          4 kB
          Vincent
        3. svm-mkl-2.png
          419 kB
          Vincent
        4. svm-mkl-1.png
          290 kB
          Vincent
        5. svm1.png
          295 kB
          Vincent
        6. svm2.png
          404 kB
          Vincent
        7. mllib svm training.png
          7 kB
          Vincent

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                VinceXie Vincent
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: