[SPARK-34833] Apply right-padding correctly for correlated subqueries

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1, 3.2.0
• Fix Version/s: 3.1.2, 3.2.0
• Component/s: SQL
• Labels: correctness
• Assignee: Takeshi Yamamuro
• Reporter: Takeshi Yamamuro
• Created: 23/Mar/21 06:57
• Updated: 25/Mar/21 13:28
• Resolved: 24/Mar/21 23:33

Description

This ticket aim at fixing the bug that does not apply right-padding for char types inside correlated subquries.
For example, a query below returns nothing in master, but a correct result is `c`.

scala> sql(s"CREATE TABLE t1(v VARCHAR(3), c CHAR(5)) USING parquet")
scala> sql(s"CREATE TABLE t2(v VARCHAR(5), c CHAR(7)) USING parquet")
scala> sql("INSERT INTO t1 VALUES ('c', 'b')")
scala> sql("INSERT INTO t2 VALUES ('a', 'b')")
scala> val df = sql("""
  |SELECT v FROM t1
  |WHERE 'a' IN (SELECT v FROM t2 WHERE t2.c = t1.c )""".stripMargin)

scala> df.show()
+---+
|  v|
+---+
+---+

This is because `ApplyCharTypePadding` does not handle the case above to apply right-padding into `'abc'`.

Issue Links

• is caused by: SPARK-33480 support char/varchar type
• links to: [Github] Pull Request #31940 (maropu)
• links to: [Github] Pull Request #31959 (cloud-fan)
• links to: [Github] Pull Request #31959 (cloud-fan)