Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-6419

Invalid unparse for VARCHAR without precision in HiveSqlDialect And SparkSqlDialect

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.37.0
    • 1.38.0
    • core

    Description

      When we execute SQL in Calcite:

      select cast(product_id as varchar) from product;
      

      Generage the HiveSQL\SparkSQL:

      select cast(product_id as varchar) from product;
      

      According to the https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-VarcharvarcharVarchar.

      In Hive, the varchar must have a precision.So when unpare VARCHAR without precision, I will convert VARCHAR to String. VARCHAR with precison do nothing.

      According to the https://spark.apache.org/docs/latest/sql-ref-datatypes.html.

      In Spark, Same as Hive. But as note, It can only be used in table schema, not functions/operators. So I will convert VARCHAR with or without precision to String;

      In SparkSQL, VarcharĀ  with precision are not effective, but no error and just a warning:

      spark-sql> select cast('value' as varchar(2));
      24/05/24 16:04:39 WARN CharVarcharUtils: The Spark cast operator does not support char/varchar type and simply treats them as string type. Please use string type directly to avoid confusion. Otherwise, you can set spark.sql.legacy.charVarcharAsString to true, so that Spark treat them as string type as same as Spark 3.0 and earlier
      value
      Time taken: 2.797 seconds, Fetched 1 row(s)

      Attachments

        Issue Links

          Activity

            People

              nobigo xiong duan
              nobigo xiong duan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: