[SPARK-33790] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.0.2, 3.1.1, 3.2.0
• Component/s: Spark Core
• Assignee: dzcxzl
• Reporter: dzcxzl
• Created: 15/Dec/20 12:04
• Updated: 15/Jan/21 16:28
• Resolved: 17/Dec/20 02:38

Description

FsHistoryProvider#checkForLogs already has FileStatus when constructing SingleFileEventLogFileReader, and there is no need to get the FileStatus again when SingleFileEventLogFileReader#fileSizeForLastIndex.
This can reduce a lot of rpc calls and improve the speed of the history server.

Issue Links

• links to: [Github] Pull Request #30780 (cxzl25)
• links to: [Github] Pull Request #30814 (dongjoon-hyun)
• links to: [Github] Pull Request #31186 (HeartSaVioR)
• links to: [Github] Pull Request #31187 (HeartSaVioR)
• links to: [Github] Pull Request #31187 (HeartSaVioR)