[SPARK-39388] Reuse orcSchema when push down Orc predicates

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 06/Jun/22 09:24
• Updated: 27/Jun/22 22:58
• Resolved: 27/Jun/22 22:58

Description

`OrcUtils.readCatalystSchema` method will trigger a file IO to read `orcSchema` when Orc pushes down predicates.

After SPARK-37463:

•  For `OrcFileFormat`, we can reuse [a read `orcSchema`](https://github.com/apache/spark/blob/cc0bf563b8caea21da5692f05e34b5f77e002ab9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala#L146) to save this file read IO when Orc pushes down predicates

• For `OrcPartitionReaderFactory`  we can achieve the same goal by moving `OrcFile.createReader` before `pushDownPredicates`.

Issue Links

• links to: https://github.com/apache/spark/pull/36774