Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24631

Cannot up cast column from bigint to smallint as it may truncate

    Details

    • Type: New JIRA Project
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.1
    • Fix Version/s: None
    • Component/s: Spark Core, Spark Submit
    • Labels:
      None

      Description

      Getting the below error when executing the simple select query,

      Sample:

      Table Description:

      name: String, id: BigInt

      val df=spark.sql("select name,id from testtable")

      ERROR: Cannot up cast column "id" from bigint to smallint as it may truncate.

      I am not doing any transformation's, I am just trying to query a table ,But still I am getting the error.

      I am getting this error only on production cluster and only for a single table, other tables are running fine.

      + more data,

      val df=spark.sql("select* from table_name")

      I am just trying this query a table. But with other tables it is running fine.

      18/06/22 01:36:29 ERROR Driver1: [] [main] Exception occurred: org.apache.spark.sql.AnalysisException: Cannot up cast `column_name` from bigint to column_name#2525: smallint as it may truncate.

      that specific column is having Bigint datatype, But there were other table's that ran fine with Bigint columns.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              HadoopSiva Sivakumar
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: