[SPARK-30410] Calculating size of table having large number of partitions causes flooding logs

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.4.5, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: SQL
• Assignee: Zhenhua Wang
• Reporter: Zhenhua Wang
• Created: 02/Jan/20 15:32
• Updated: 09/Jan/20 09:10
• Resolved: 08/Jan/20 14:23

Description

For a partitioned table, if the number of partitions are very large, e.g. tens of thousands or even larger, calculating its total size causes flooding logs.

The flooding happens in:

1. `calculateLocationSize` prints the starting and ending for calculating the location size, and it is called per partition;
2. `bulkListLeafFiles` prints all partition paths.

Issue Links

• links to: GitHub Pull Request #27079
• links to: GitHub Pull Request #27143