Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33879

Char Varchar values fails w/ match error as partition columns

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • SQL
    • None

    Description

      spark-sql> select * from t10 where c0='abcd';
      20/12/22 15:43:38 ERROR SparkSQLDriver: Failed in [select * from t10 where c0='abcd']
      scala.MatchError: CharType(10) (of class org.apache.spark.sql.types.CharType)
      	at org.apache.spark.sql.catalyst.expressions.CastBase.cast(Cast.scala:815)
      	at org.apache.spark.sql.catalyst.expressions.CastBase.cast$lzycompute(Cast.scala:842)
      	at org.apache.spark.sql.catalyst.expressions.CastBase.cast(Cast.scala:842)
      	at org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeEval(Cast.scala:844)
      	at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:476)
      	at org.apache.spark.sql.catalyst.catalog.CatalogTablePartition.$anonfun$toRow$2(interface.scala:164)
      	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
      	at scala.collection.Iterator.foreach(Iterator.scala:941)
      	at scala.collection.Iterator.foreach$(Iterator.scala:941)
      	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
      	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
      	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
      	at org.apache.spark.sql.types.StructType.foreach(StructType.scala:102)
      	at scala.collection.TraversableLike.map(TraversableLike.scala:238)
      	at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
      	at org.apache.spark.sql.types.StructType.map(StructType.scala:102)
      	at org.apache.spark.sql.catalyst.catalog.CatalogTablePartition.toRow(interface.scala:158)
      	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.$anonfun$prunePartitionsByFilter$3(ExternalCatalogUtils.scala:157)
      	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.$anonfun$prunePartitionsByFilter$3$adapted(ExternalCatalogUtils.scala:156)
      

      c0 is a partition column, it fails in partition pruning rule

      Attachments

        Activity

          People

            Qin Yao Kent Yao
            Qin Yao Kent Yao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: