[SPARK-44634] Encoders.bean does no longer support nested beans with type arguments

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1, 3.5.0, 4.0.0
• Fix Version/s: 3.4.2, 3.5.0, 4.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Giambattista Bloisi
• Created: 02/Aug/23 09:13
• Updated: 15/Dec/23 04:52
• Resolved: 07/Aug/23 14:14

Description

Hi,

  while upgrading a project from spark 2.4.0 to 3.4.1 version, I have encountered the same problem described in java - Encoders.bean attempts to check the validity of a return type considering its generic type and not its concrete class, with Spark 3.4.0 - Stack Overflow.

Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when the passed class contains a field whose type is a nested bean with type arguments:

 

class A<T> {
   T value;
   // value getter and setter
}

class B {
   A<String> stringHolder;
   // stringHolder getter and setter
}

Encoders.bean(B.class); // throws "SparkUnsupportedOperationException: [ENCODER_NOT_FOUND]..."

 

 

It looks like this is a regression introduced with SPARK-42093 SQL Move JavaTypeInference to AgnosticEncoders while getting rid of TypeToken, that somehow managed that case.

Issue Links

• Blocked: SPARK-45311 Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"
• links to: GitHub Pull Request #42327
• links to: GitHub Pull Request #42379