[SPARK-32331] Keep advanced statistics when pruning partitions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Peter Toth
• Created: 16/Jul/20 13:18
• Updated: 18/Jan/21 09:15
• Resolved: 18/Jan/21 09:15

Description

The rule PruneFileSourcePartitions deletes the advances statistics (cardinality, column stats) of the underlying LogicalRelation and keeps only sizeInBytes as that is the only value we can be sure that is accurate.
I think we should keep all statistics as they are a good upper limit estimates and by keeping them other rules that depend on the presence of these statistics (like CostBasedJoinReorder) can still ran.

Issue Links

• duplicates: SPARK-34119 Keep necessary stats after partition pruning
• links to: [Github] Pull Request #29132 (peter-toth)
• links to: [Github] Pull Request #29132 (peter-toth)