Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14536

NPE in JDBCRDD when array column contains nulls (postgresql)

    Details

      Description

      At https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala#L453 it is assumed that the JDBC driver will definitely return a non-null `Array` object from the call to `getArray`, and that in the event of a null array it will return an non-null `Array` object with a null underlying array. But as you can see here https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/PgResultSet.java#L387 that isn't the case, at least for PostgreSQL. This causes a `NullPointerException` whenever an array column contains null values. It seems like the PostgreSQL JDBC driver is probably doing the wrong thing, but even so there should be a null check in JDBCRDD. I'm happy to submit a PR if that would be helpful.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tsuresh Suresh Thalamati
                Reporter:
                jeremyrsmith Jeremy Smith
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: