[SPARK-12266] cannot handle postgis raster type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.5.2
• Component/s: Input/Output
• Labels: dataframe postgres sql

Environment

PostGIS Server:
Ubuntu-14.04
Postgres 9.4.5
PostGIS 2.2.1
Spark standalone client:
Java 7
Spark 1.5.2
PostGIS JDBC driver built within java directory of PostGIS 2.1.8, see: http://postgis.net/docs/manual-2.1/postgis_installation.html#idp58575520

• Assignee: Unassigned
• Reporter: Severin Thaler
• Created: 10/Dec/15 18:44
• Updated: 11/Dec/15 17:30
• Resolved: 10/Dec/15 19:00

Description

on server postgis running with a table that has a column of type raster
on client running spark standalone application that pulls data from postgis server using and jdbc driver and spark-sql:

val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val jdbcDF = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:postgresql://postgishost/test?user=postgres&password=userpassword", "dbtable" -> "atlas")).load()
val rows = jdbcDF.take(3)
println(rows(0).toString())
println(rows(1).toString())
println(rows(2).toString())

when running this spark app i get:

java.sql.SQLException: Unsupported type 1111
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:103)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:140)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:140)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:139)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:91)
  at org.apache.spark.sql.execution.datasources.jdbc.DefaultSource.createRelation(DefaultSource.scala:60)
  at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:125)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114)
  at rdd.BasicRDDTests$$anonfun$1.apply$mcV$sp(BasicRDDTests.scala:65)

Issue Links

• duplicates: SPARK-10186 Add support for more postgres column types