[SPARK-49160] make JdbcUtils methods makeGetter and makeSetter public to extend custom dialect support for complex types

• Type: Wish
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Spark Core, SQL
• Assignee: Unassigned
• Reporter: Maksim Lixakov
• Created: 08/Aug/24 09:57
• Updated: 08/Aug/24 12:15

Description

Currently, in Apache Spark's JdbcUtils (across all versions, including Spark 4), there is no support for complex types like StructType, MapType, except for ArrayType. This limitation is evident in the private methods makeGetter and makeSetter.

As Spark provides the ability to implement custom dialects, users cannot implement custom handling for these complex types because the methods are private and they encounter the following error:

 

org.apache.spark.SparkSQLException: Unsupported type <complex_type>.

The suggestion is to change the way these methods are called. Specifically, the getter/setter methods should be able to be provided by the `JdbcDialect`, rather than only being hardcoded in `JdbcUtils`.

I suppose it can be done in one of two ways:

1. Modify the makeGetter and makeSetter methods to delegate to the dialect

    
    class JdbcUtils {
      def makeGetter(args): Option[Getter] = {
        dialect.makeGetter(args).orElse {
          ...
        }
      }
    
      def makeSetter(args): Option[Setter] = {
        dialect.makeSetter(args).orElse {
          ...
        }
      }
    }
   

2. Move the makeGetter/makeSetter methods out of JdbcUtils into JdbcDialect, allowing custom dialects to override them.

    
    abstract class JdbcDialect {
      def makeGetter(args): Option[Getter]
      def makeSetter(args): Option[Setter]
    }
    
    object JdbcUtils {
      def getGetter(args): Option[Getter] = {
        // logic to fetch getter from JdbcDialect
      }
    
      def getSetter(args): Option[Setter] = {
        // logic to fetch setter from JdbcDialect
      }
    }
   

Sub-Tasks

• Add support for StructType and MapType in JdbcUtils: Open