[ARROW-8902] [rust][datafusion] optimize count(*) queries on parquet sources

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Invalid
• Component/s: Rust, Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/25036
• Assignee: Unassigned
• Reporter: Alex Gaynor
• Created: 23/May/20 00:02
• Updated: 11/Jan/23 08:03
• Resolved: 26/Apr/21 13:18

Description

Currently, as far as I can tell, when you perform a `select count from dataset` in datafusion against a parquet dataset, the way this is implemented is by doing a scan on column 0, and counting up all of the rows (specifically I think it counts the # of rows in each batch).

 

However, for the specific case of just counting everythign in a parquet file, you can just read the rowcount from the footer metadata, so it's O(1) instead of O