[SPARK-24797] Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.3.0, 2.3.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Nan Zhu
• Created: 13/Jul/18 02:44
• Updated: 13/Jul/18 17:14
• Resolved: 13/Jul/18 17:14

Description

the current code path ignore the value of spark.sql.hive.convertMetastoreParquet when building data source table 

 

https://github.com/apache/spark/blob/e0559f238009e02c40f65678fec691c07904e8c0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L263

 

as a result, even I turned off spark.sql.hive.convertMetastoreParquet, Spark SQL still uses its own parquet reader to access table instead of delegate to serder

Issue Links

• links to: [Github] Pull Request #21757 (CodingCat)