[SPARK-21997] Spark shows different results on char/varchar columns on Parquet

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.2, 2.1.1, 2.2.0, 2.3.4, 2.4.4, 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 13/Sep/17 18:14
• Updated: 13/Jan/20 20:44

Description

SPARK-19459 resolves CHAR/VARCHAR issues in general, but Spark shows different results according to the SQL configuration, spark.sql.hive.convertMetastoreParquet. We had better fix this. Actually, the default of `spark.sql.hive.convertMetastoreParquet` is true, so the result is wrong by default.

scala> sql("CREATE TABLE t_char(a CHAR(10), b VARCHAR(10)) STORED AS parquet")
scala> sql("INSERT INTO TABLE t_char SELECT 'a', 'b'")
scala> sql("SELECT * FROM t_char").show
+---+---+
|  a|  b|
+---+---+
|  a|  b|
+---+---+

scala> sql("set spark.sql.hive.convertMetastoreParquet=false")

scala> sql("SELECT * FROM t_char").show
+----------+---+
|         a|  b|
+----------+---+
|a         |  b|
+----------+---+

Issue Links

• relates to: SPARK-19459 ORC tables cannot be read when they contain char/varchar columns
• links to: [Github] Pull Request #19235 (dongjoon-hyun)