[SPARK-8756] Keep cached information and avoid re-calculating footers in ParquetRelation2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Shepherd: Cheng Lian
• Created: 01/Jul/15 09:52
• Updated: 24/Jul/15 09:42
• Resolved: 24/Jul/15 09:41

Description

Currently, in ParquetRelation2, footers are re-read every time refresh() is called. But we can check if it is possibly changed before we do the reading because reading all footers will be expensive when there are too many partitions.

Issue Links

• links to: [Github] Pull Request #7154 (viirya)