[SPARK-40600] Support recursiveFileLookup for partitioned datasource

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: SQL

Environment

Spark: 3.1.1

Hive: 3.1.2

• Assignee: Unassigned
• Reporter: Zhen Wang
• Created: 28/Sep/22 13:12
• Updated: 30/Sep/22 01:28

Description

I use hive tez engine to execute union statement and insert into partitioned table may generate HIVE_UNION_SUBDIR subdirectory, and when I use spark sql to read this partitioned table, the data below HIVE_UNION_SUBDIR is not read.

For non-partitioned table, I can read the subdirectories of the table when setting recursiveFileLookup to true, but for partitioned table, it seems impossible to set recursiveFileLookup to true.

So I want to support recursiveFileLookup for partitioned table.

Issue Links

• relates to: SPARK-28098 Native ORC reader doesn't support subdirectories with Hive tables
• links to: [Github] Pull Request #38053 (wForget)