[SPARK-21974] SVD computation results in failure to load NativeSystemARPACK and NativeRefARPACK

• Type: Wish
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 2.2.0
• Component/s: MLlib
• Assignee: Unassigned
• Reporter: Aleksandr Ovcharenko
• Created: 11/Sep/17 10:26
• Updated: 11/Sep/17 13:15
• Resolved: 11/Sep/17 11:37

Description

Hello guys,

I am trying to implement the SVD computation with Spark 2.2.0 using the Python interface. However I keep running into the following issues:
17/09/11 11:41:26 WARN ARPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemARPACK
17/09/11 11:41:26 WARN ARPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefARPACK

I've tried to install ARPACK myself and set LD_LIBRARY_PATH accordingly before running pyspark. I've also trid to pass the "--packages com.github.fommil.netlib:all:1.1.2" command in order to pick up netlib-java correctly.

Is there anything I'm missing? Is it possible to see why native ARPACK is not being picked up when setting the LD_LIBRARY_PATH correctly? Could you advise something to get rid of either or both warnings? I'm really getting a huge performance hit because of that.

Thank you, I appreciate your response.