[SPARK-11827] Support java.math.BigInteger in Type-Inference utilities for POJOs

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Fix Version/s: 2.0.0
• Component/s: SQL
• Assignee: kevin yu
• Reporter: Abhilash Pallerlamudi
• Created: 18/Nov/15 20:56
• Updated: 20/May/16 21:15
• Resolved: 20/May/16 04:43

Description

I get the below exception when creating DataFrame using RDD of JavaBean having a property of type java.math.BigInteger

scala.MatchError: class java.math.BigInteger (of class java.lang.Class)
at org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1182)
at org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1181)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:1181)
at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:419)
at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:447)

I don't see the support for java.math.BigInteger in org.apache.spark.sql.catalyst.JavaTypeInference.scala

Issue Links

• relates to: SPARK-15445 Build fails for java 1.7 after adding java.math.BigInteger support [SPARK-11827][SQL]
• links to: [Github] Pull Request #10125 (kevinyu98)
• links to: [Github] Pull Request #13233 (ted-yu)