[SPARK-24631] Cannot up cast column from bigint to smallint as it may truncate

• Type: New JIRA Project
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.1
• Component/s: Spark Core, Spark Submit
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Sivakumar
• Created: 22/Jun/18 15:17
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/19 05:43

Description

Getting the below error when executing the simple select query,

Sample:

Table Description:

name: String, id: BigInt

val df=spark.sql("select name,id from testtable")

ERROR: Cannot up cast column "id" from bigint to smallint as it may truncate.

I am not doing any transformation's, I am just trying to query a table ,But still I am getting the error.

I am getting this error only on production cluster and only for a single table, other tables are running fine.

+ more data,

val df=spark.sql("select* from table_name")

I am just trying this query a table. But with other tables it is running fine.

18/06/22 01:36:29 ERROR Driver1: [] [main] Exception occurred: org.apache.spark.sql.AnalysisException: Cannot up cast `column_name` from bigint to column_name#2525: smallint as it may truncate.

that specific column is having Bigint datatype, But there were other table's that ran fine with Bigint columns.