[ZEPPELIN-3109] pyspark can't access mariadb via jdbc

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.3
• Component/s: Interpreters, pySpark, python-interpreter

Environment

host OS: centos7
docker: ver17.09-ce
container: apache/zeppelin:0.7.3

• Assignee: Unassigned
• Reporter: shota suzuki
• Created: 18/Dec/17 07:39
• Updated: 18/Dec/17 07:39

Description

I want to access mariadb using pyspark but error raised.
my notebook as follows.

```bash
%spark.dep
z.load("/zeppelin/interpreter/jdbc/mariadb-java-client-2.2.0.jar")
->
res0: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Dependency@6ed2e421
```

```bash
%spark.pyspark
import pyspark
print(pyspark._version_)

dataframe = sqlContext.read.format("jdbc").options(
url ="jdbc:mariadb://mariadb:3306/db",
driver="org.mariadb.jdbc.Driver",
dbtable="table",
user="user",
password="pass"
).load().take(10)

->
2.1.0

Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-400087723623135418.py", line 367, in <module>
raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-400087723623135418.py", line 360, in <module>
exec(code, _zcUserQueryNameSpace)
File "<stdin>", line 8, in <module>
File "/zeppelin/interpreter/spark/pyspark/pyspark.zip/pyspark/sql/readwriter.py", line 155, in load
return self._df(self._jreader.load())
File "/zeppelin/interpreter/spark/pyspark/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in _call_
answer, self.gateway_client, self.target_id, self.name)
File "/zeppelin/interpreter/spark/pyspark/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/zeppelin/interpreter/spark/pyspark/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o47.load.
: java.lang.UnsatisfiedLinkError: Unable to load library 'c': /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:145)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:188)
at com.sun.jna.Library$Handler.<init>(Library.java:123)
at com.sun.jna.Native.loadLibrary(Native.java:255)
at com.sun.jna.Native.loadLibrary(Native.java:241)
at org.mariadb.jdbc.internal.util.PidFactory$CLibrary.<clinit>(PidFactory.java:112)
at org.mariadb.jdbc.internal.util.PidFactory.lambda$getInstance$0(PidFactory.java:78)
at org.mariadb.jdbc.internal.com.send.SendHandshakeResponsePacket.writeConnectAttributes(SendHandshakeResponsePacket.java:211)
at org.mariadb.jdbc.internal.com.send.SendHandshakeResponsePacket.send(SendHandshakeResponsePacket.java:184)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:796)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:775)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:453)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1084)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:493)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:156)
at org.mariadb.jdbc.Driver.connect(Driver.java:90)
at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:59)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:50)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:113)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:45)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
```

mariadb and zeppelin container are reachable. I confirmed it by jdbc interpreter.