[SPARK-32432] Add support for reading ORC/Parquet files with SymlinkTextInputFormat

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Noritaka Sekiyama
• Created: 24/Jul/20 16:43
• Updated: 04/Mar/22 16:34

Description

Hive style symlink (SymlinkTextInputFormat) is commonly used in different analytic engines including prestodb and prestosql.

Currently SymlinkTextInputFormat works with JSON/CSV files but does not work with ORC/Parquet files in Apache Spark (and Apache Hive).

On the other hand, prestodb and prestosql support SymlinkTextInputFormat with ORC/Parquet files.

This issue is to add support for reading ORC/Parquet files with SymlinkTextInputFormat in Apache Spark.

 

Related links

• Hive's SymlinkTextInputFormat: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java
• prestosql's implementation to add support for reading avro files with SymlinkTextInputFormat: https://github.com/vincentpoon/prestosql/blob/master/presto-hive/src/main/java/io/prestosql/plugin/hive/BackgroundHiveSplitLoader.java

Issue Links

• links to: [Github] Pull Request #29330 (moomindani)
• links to: [Github] Pull Request #29330 (moomindani)
• links to: [Github] Pull Request #35734 (CHENXCHEN)