[SPARK-37643] when charVarcharAsString is true, char datatype partition table query incorrect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2, 3.2.0
• Fix Version/s: 3.1.3, 3.3.0, 3.2.2
• Component/s: SQL
• Labels: correctness

Environment

spark 3.2.0

• Assignee: YuanGuanhu
• Reporter: YuanGuanhu
• Created: 14/Dec/21 09:30
• Updated: 08/Jul/22 01:49
• Resolved: 18/Apr/22 15:20

Description

This ticket aim at fixing the bug that does not apply right-padding for char types column when charVarcharAsString is true and partition data length is less than defined length.
For example, a query below returns nothing in master, but a correct result is `abc`.

scala> sql("set spark.sql.legacy.charVarcharAsString=true")
scala> sql("CREATE TABLE tb01(i string, c char(5)) USING parquet partitioned by (c)")
scala> sql("INSERT INTO tb01 values(1, 'abc')")
scala> sql("select c from tb01 where c = 'abc'").show
+---+
|  c|
+---+
+---+

This is because `ApplyCharTypePadding` rpad the expr to charLength. We should handle this consider conf spark.sql.legacy.charVarcharAsString value.

Issue Links

• links to: [Github] Pull Request #34900 (fhygh)
• links to: [Github] Pull Request #36187 (fhygh)