[SPARK-49161] Add support for StructType and MapType in JdbcUtils

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Spark Core, SQL
• Assignee: Unassigned
• Reporter: Maksim Lixakov
• Created: 08/Aug/24 10:39
• Updated: 08/Aug/24 12:02

Description

Currently, in Apache Spark's JdbcUtils (across all versions, including Spark 4), there is no support for StructType, MapType.

The current implementation in JdbcUtils supports ArrayType

To extend the functionality and enable users to read and write StructType and MapType from their databases analog, it is proposed to add similar handling for these complex types in the makeGetter and makeSetter methods.

For MapType it might look like:

case MapType(keyType, valueType, valueContainsNull) =>
  (rs: ResultSet, row: InternalRow, pos: Int) =>
    val mapData = rs.getObject(pos + 1).asInstanceOf[Map[Any, Any]]
    if (mapData == null) {
      row.setNullAt(pos)
    } else {
      val keys = mapData.keys.map(getCatalystType(keyType).getJavaValue).toArray
      val values = mapData.values.map(getCatalystType(valueType).getJavaValue).toArray
      row.update(pos, new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values)))
    }