[SPARK-41007] BigInteger Serialization doesn't work with JavaBean Encoder

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.4.0
• Component/s: Java API
• Assignee: Daniel Fiterma
• Reporter: Daniel Fiterma
• Created: 03/Nov/22 15:07
• Updated: 08/Nov/22 01:34
• Resolved: 08/Nov/22 01:34

Description

When creating a dataset using the Java Bean Encoder with a bean that contains a field which is a java.math.BigInteger the dataset will fail to serialize correctly. When trying to serialize the dataset, Spark throws the following error:

 

Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot up cast `bigInteger` from struct<> to decimal(38,18).
 

 

Reproduction steps:

Using the Java Dataset API:

1. Create a Bean with a  java.math.BigInteger field
2. Pass said Bean into the Java SparkSession createDataset function

 

Issue Links

• links to: [Github] Pull Request #38500 (dfit99)