[SPARK-30516] statistic estimation of FileScan should take partitionFilters and partition number into account

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Hu Fuwang
• Created: 15/Jan/20 06:39
• Updated: 30/Apr/20 16:49

Description

Currently, FileScan.estimateStatistics does not take partitionFilters and partition number into account, which may lead to bigger sizeInBytes. It should be reasonable to change it to involve partitionFilters and partition number when estimating the statistics.

Issue Links

• links to: GitHub Pull Request #27213