[SPARK-12048] JDBCRDD calls close() twice - SQLite then throws an exception

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Fix Version/s: 1.5.3, 1.6.0, 2.0.0
• Component/s: SQL
• Labels: starter
• External issue URL: http://stackoverflow.com/questions/32860100/sqlite-error-connection-is-closed-when-connecting-from-spark-via-jdbc-to-sqlite
• Assignee: R. H.
• Reporter: R. H.
• Created: 30/Nov/15 13:21
• Updated: 24/Dec/15 13:26
• Resolved: 06/Dec/15 16:28

Description

The following code works:

val tableData = sqlContext.read.format("jdbc")
.options(
Map(
"url" -> "jdbc:sqlite:/tmp/test.db",
"dbtable" -> "testtable")).load()

but an exception gets reported. From the log:

15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection

15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)

So Spark succeeded to close the JDBC connection, and then it fails to close the JDBC statement.

Looking at the source, close() seems to be called twice:

context.addTaskCompletionListener{ context => close() }

and in

def hasNext

If you look at the close() method (around line 443)

def close() {
if (closed) return

you can see that it checks the variable closed, but that value is never set to true.

So a trivial fix should be to set "closed = true" at the end of close().

Issue Links

• links to: [Github] Pull Request #10101 (rh99)
• links to: [Github] Pull Request #10320 (tedyu)