[SPARK-21791] ORC should support column names with dot

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 20/Aug/17 03:39
• Updated: 02/May/21 01:37
• Resolved: 10/Sep/17 00:54

Description

PARQUET

scala> Seq(Some(1), None).toDF("col.dots").write.parquet("/tmp/parquet_dot")
scala> spark.read.parquet("/tmp/parquet_dot").show
+--------+
|col.dots|
+--------+
|       1|
|    null|
+--------+

ORC

scala> Seq(Some(1), None).toDF("col.dots").write.orc("/tmp/orc_dot")
scala> spark.read.orc("/tmp/orc_dot").show
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '.' expecting ':'(line 1, pos 10)

== SQL ==
struct<col.dots:int>
----------^^^

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• is related to: SPARK-35274 old hive table's all columns are read when column pruning applies in spark3.0
• is superceded by: SPARK-20682 Add new ORCFileFormat based on Apache ORC
• links to: [Github] Pull Request #19651 (dongjoon-hyun)
• links to: PR