[SPARK-34960] Aggregate (Min/Max/Count) push down for ORC

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 05/Apr/21 20:00
• Updated: 22/Apr/22 01:20
• Resolved: 29/Oct/21 00:29

Description

Similar to Parquet (https://issues.apache.org/jira/browse/SPARK-34952), we can also push down certain aggregations into ORC. ORC exposes column statistics in interface `org.apache.orc.Reader` (https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L118 ), where Spark can utilize for aggregation push down.

Issue Links

• depends upon: SPARK-34952 DS V2 Aggregate push down
• links to: [Github] Pull Request #34298 (c21)
• links to: [Github] Pull Request #36311 (c21)
• links to: [Github] Pull Request #36311 (c21)