[SPARK-36513] Using openGauss ( like postgres) as Spark Datasource, Spark thorws cast errors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.3.0
• Component/s: Java API, SQL

Environment

Spark 3.3.0

Scala 2.12

• Assignee: Unassigned
• Reporter: Xiansen Chen
• Created: 14/Aug/21 14:47
• Updated: 12/Dec/22 18:11
• Resolved: 15/Aug/21 03:10

Description

Hi, I'm trying to use openGauss as the Spark Datasource and implementing some datasource interfaces. I would like to know if there is any solution. Thank you!

 

But when I try to get the column which is VARCHAR type, I got an error like this:

When I use getInt() or getDouble() for other type, the code  is working fine.

And here is some code:

 

object OpenGaussTable {
  /*Table products*/
  val schema: StructType = new StructType().add("name", StringType)

}
...

class OpenGaussPartitionReader(connectionProperties: ConnectionProperties) extends PartitionReader[InternalRow] {
  private val connection = DriverManager.getConnection(
    connectionProperties.url, connectionProperties.user, connectionProperties.password
  )
  private val statement = connection.createStatement()
  private val resultSet = statement.executeQuery(s"select * from ${connectionProperties.tableName}")

  override def next(): Boolean = resultSet.next()

  override def get(): InternalRow = InternalRow(resultSet.getString(1))

  override def close(): Unit = connection.close()
}

 

the complete code address：

https://pastebin.ubuntu.com/p/vCBssjhc2r/

https://pastebin.ubuntu.com/p/jCcDbb79NG/