Getting the below error when executing the simple select query,
name: String, id: BigInt
val df=spark.sql("select name,id from testtable")
ERROR: Cannot up cast column "id" from bigint to smallint as it may truncate.
I am not doing any transformation's, I am just trying to query a table ,But still I am getting the error.
I am getting this error only on production cluster and only for a single table, other tables are running fine.
+ more data,
val df=spark.sql("select* from table_name")
I am just trying this query a table. But with other tables it is running fine.
18/06/22 01:36:29 ERROR Driver1:  [main] Exception occurred: org.apache.spark.sql.AnalysisException: Cannot up cast `column_name` from bigint to column_name#2525: smallint as it may truncate.
that specific column is having Bigint datatype, But there were other table's that ran fine with Bigint columns.