[SPARK-49960] Support end DeveloperAPI provided AgnosticEncoders based on ExpressionEncoders

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SQL
• Labels: Encoding pull-request-available
• Target Version/s: 4.0.0
• Flags: Patch

Language

Scala

• Assignee: Unassigned
• Reporter: Chris Twiner
• Created: 14/Oct/24 16:17
• Updated: 16/Oct/24 16:01

Description

In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used throughout the original api.  SPARK-49025 removed re-use of ExpressionEncoders for derivation of tuples rendering user provided ExpressionEncoders such as Frameless incompatible with joinWith (which only uses the AgnosticEncoder hierarchy).  There are other likely combinations where the encoder is not passed into the function.

This patch aims to add an extension AgnosticExpressionPathEncoder trait with added:

def toCatalyst(input: Expression): Expression

def fromCatalyst(inputPath: Expression): Expression 

functions mirroring the underlying Des/SerializerBuildHelper approach (also used by Frameless).  In turn the BuildHelper's delegate to these functions when the AgnosticEncoder is an AgnosticExpressionPathEncoder.

Issue Links

• links to: GitHub Pull Request #48477