[SPARK-7697] Column with an unsigned int should be treated as long in JDBCRDD

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: DAITO Teppei
• Created: 18/May/15 02:53
• Updated: 27/May/15 14:05
• Resolved: 27/May/15 07:28

Description

Columns with an unsigned numeric type in JDBC should be treated as the next 'larger' Java type
in JDBCRDD#getCatalystType .
https://github.com/apache/spark/blob/517eb37a85e0a28820bcfd5d98c50d02df6521c6/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JDBCRDD.scala#L49

q.sql

create table t1 (id int unsigned);
insert into t1 values (4234567890);

T1.scala

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext

object T1 {
  def main(args: Array[String]) {
    val sc = new SparkContext(new SparkConf())
    val s = new SQLContext(sc)
    val url = "jdbc:mysql://localhost/test"
    val t1 = s.jdbc(url, "t1")
    t1.printSchema()
    t1.collect().foreach(println)
  }
}

This code caused error like below.

15/05/18 11:39:51 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, xxx): com.mysql.jdbc.exceptions.jdbc4.MySQLDataException: '4.23456789E9' in column '1' is outside valid range for the datatype INTEGER.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
        at com.mysql.jdbc.Util.getInstance(Util.java:360)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870)
        at com.mysql.jdbc.ResultSetImpl.throwRangeException(ResultSetImpl.java:7090)
        at com.mysql.jdbc.ResultSetImpl.parseIntAsDouble(ResultSetImpl.java:6364)
        at com.mysql.jdbc.ResultSetImpl.getInt(ResultSetImpl.java:2484)
        at org.apache.spark.sql.jdbc.JDBCRDD$$anon$1.getNext(JDBCRDD.scala:344)
        at org.apache.spark.sql.jdbc.JDBCRDD$$anon$1.hasNext(JDBCRDD.scala:399)
...

Issue Links

• relates to: SPARK-7897 Column with an unsigned bigint should be treated as DecimalType in JDBCRDD
• links to: [Github] Pull Request #6229 (viirya)