[SPARK-14536] NPE in JDBCRDD when array column contains nulls (postgresql)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: SQL
• Labels: NullPointerException
• Assignee: Suresh Thalamati
• Reporter: Jeremy Smith
• Created: 11/Apr/16 16:12
• Updated: 12/Dec/22 18:11
• Resolved: 21/Jan/17 03:24

Description

At https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala#L453 it is assumed that the JDBC driver will definitely return a non-null `Array` object from the call to `getArray`, and that in the event of a null array it will return an non-null `Array` object with a null underlying array. But as you can see here https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/PgResultSet.java#L387 that isn't the case, at least for PostgreSQL. This causes a `NullPointerException` whenever an array column contains null values. It seems like the PostgreSQL JDBC driver is probably doing the wrong thing, but even so there should be a null check in JDBCRDD. I'm happy to submit a PR if that would be helpful.

Issue Links

• duplicates: SPARK-8500 Support for array types in JDBCRDD
• is part of: SPARK-20859 SQL Loader does not recognize multidimensional columns in postgresql (like integer[]][])
• links to: [Github] Pull Request #15192 (sureshthalamati)
• links to: [Github] Pull Request #17460 (sureshthalamati)