[SPARK-45081] Encoders.bean does no longer work with read-only properties

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 3.4.2, 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Giambattista Bloisi
• Created: 05/Sep/23 14:42
• Updated: 15/Dec/23 04:52
• Resolved: 12/Sep/23 14:17

Description

Since Spark 3.4.x an exception is thrown when Encoders.bean is called providing a bean having read-only properties, such as:

 

public static class ReadOnlyPropertyBean implements Serializable {
    public boolean isEmpty() {
      return true;
    }
} 

 

 
Encoders.bean(ReadOnlyPropertyBean.class) will throw:

java.util.NoSuchElementException: None.get
        at scala.None$.get(Option.scala:529)
        at scala.None$.get(Option.scala:527)
        at org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$deserializerFor$8(ScalaReflection.scala:359)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor(ScalaReflection.scala:348)
        at org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor(ScalaReflection.scala:183)
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:56)
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:62)
        at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179)
        at org.apache.spark.sql.Encoders.bean(Encoders.scala) 

This problem is described also in link Encoders.bean doesn't work anymore on a Java POJO, with Spark 3.4.0

Issue Links

• Blocked: SPARK-45311 Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"
• links to: GitHub Pull Request #42829
• links to: GitHub Pull Request #42913